AI推理提速：动态调整每次生成的字数
AI生成文本时，通常一次只预测一个词，速度慢。一种叫“推测解码”的技术让轻量模型先批量预测多个词，再由大模型验证，实现无损加速。但现有方法对所有输入都用固定批量大小，而这篇论文发现：最佳批量大小因输入而异，且集中在训练值附近，形成低维决策空间。他们提出BlockPilot，在模型读完输入后，仅用一次预测就动态决定本次该批量生成多少词。实验显示，在Qwen3-4B模型上，平均接受长度达5.92个词，速度提升4.2倍，且几乎不增加额外开销。这不是你明天能直接用的功能，但它揭示了AI推理优化的新方向：让模型自己决定“一次说多少话”最划算。
📄 原文摘要(英文)
Speculative decoding accelerates inference by using a lightweight draft model to generate candidate tokens in parallel, and are then verified by the target model, enabling lossless acceleration. Recently, diffusion-based speculative decoding further improves parallelism by generating multiple tokens per forward pass via block-level diffusion, achieving state-of-the-art (SOTA) performance. However, existing methods adopt a fixed inference block size and assume a uniform optimal decoding strategy across all inputs. In this paper, we show that this assumption is suboptimal, as the optimal block size varies across samples and plays a critical role in speculative decoding performance. Moreover, these values exhibit a clear local structure, concentrating around the training block size, which reduces the problem to a low-dimensional and structured decision space. Based on these insights, we propose BlockPilot, a sample-adaptive policy that predicts the optimal block size from the prefilling representation. Specifically, we formulate block size selection as a lightweight policy learning problem and propose an instance-adaptive decision mechanism that predicts the optimal block size based on the representation of the prefilling stage. The prediction is performed only once after prefilling, allowing for seamless integration. Extensive experiments demonstrate that our method is plug-and-play, introduces minimal overhead, and consistently improves efficiency, achieving an acceptance length of 5.92 and a 4.20times speedup on Qwen3-4B under temperature T=1.