号称无损的AI加速，其实只有一半概率真的无损
「无损」这个词在 AI 论文里经常被用得很松。这篇论文拆了一个叫 Orthrus 的加速方案——它号称能在不改变模型输出结果的前提下，让生成速度快好几倍。作者自己复现了一遍，发现所谓「无损」只在一种高精度计算下成立；换成现在主流显卡默认的 BF16 精度，只有 45% 的输入能保证输出和原模型一字不差。但有意思的是，输出虽然对不上，下游任务的表现却几乎不受影响。也就是说，「输出完全一致」和「效果不差」是两回事，很多论文拿前者当卖点，实际用户在乎的其实是后者。
📄 原文摘要(英文)
Orthrus is a hybrid autoregressive-diffusion architecture that accelerates autoregressive language-model inference by generating multiple tokens in parallel while using a frozen autoregressive backbone. Its central claim is that an intra-model consensus mechanism enables lossless speculative decoding, producing the same output sequence as the autoregressive model. We independently reproduce Orthrus and examine this claim under different numerical precisions. Under BF16 inference, exact trajectory matching occurs in only 45% of cases for the authors' checkpoint and 43% for our independently trained model across 1,190 prompts from 12 domains. The probability of exact matching is also strongly associated with the response-conditional perplexity of the reference model. Despite this trajectory divergence, Orthrus does not show systematic degradation on downstream lm-eval-harness benchmarks. In contrast, repeating the trajectory evaluation with FP32 yields exact trajectory matching on all evaluated prompts. These results show that the practical losslessness of Orthrus depends on numerical precision and that exact trajectory equivalence should be evaluated separately from downstream task performance.