训练和推理对不上，也能忍
让 AI 学习时，通常得先让它自己答题、再由老师批改；而生成答案很贵，于是工程上常用量化推理和异步生成来省钱，代价是「出题的那个模型」和「被训练的那个模型」对不上。这篇测的就是这种错位能忍多少：在三个师生组合上把量化、权重滞后、人为噪声都放到同一把尺子上量，发现蒸馏能扛住相当程度的错位而不掉质量，超过某个经验预算才开始退化；同样的错位水平放到强化学习里早就不稳了，蒸馏却还站得住。他们还发现错位会改变生成时走到的前缀，这种错误没法靠逐 token 修正补回来，并给了一个轻量准入检查，不必跑完整训练就能筛掉超预算的配置。落到工程上：按模型定制的 FP8 配方保住训练质量，KV 缓存能装下至少 1.93 倍的 token；FP8 加异步生成，比同步 BF16 早 1.62 倍达到同等质量。
📄 原文摘要(英文)
On-policy distillation (OPD) trains a student language model on its own responses, guided by a fixed teacher. Generating these responses is expensive, so systems use quantized inference and asynchronous rollouts to reduce the cost. Both, however, introduce training-inference mismatch between the rollout engine and the model being trained. Such mismatch can destabilize reinforcement learning (RL), but its effects on OPD remain unclear. This paper investigates how much mismatch OPD tolerates, where its effects arise, and how training systems can use this tolerance. Measuring all mismatch sources on one scale, we examine quantization across three student-teacher pairs, with additional tests of weight staleness and injected noise on one pair. OPD tolerates substantial mismatch without meaningful loss of training quality but degrades at higher levels, revealing an empirical mismatch budget. Both the quality cost of mismatch and the mismatch introduced by quantization depend on the model, with some layers particularly sensitive to reduced precision. In controlled comparisons, OPD remains stable under rollout configurations that destabilize RL methods. Under weight noise, mismatch also shifts the prefixes reached during generation, introducing errors that per-token corrections cannot address. To put these findings into practice, we introduce a lightweight admission check that screens rollout configurations against a calibrated budget without requiring a full training run for each candidate. Model-specific FP8 recipes preserve training quality while providing capacity for at least 1.93times as many KV-cache tokens as BF16. Combining FP8 with asynchronous rollouts reaches the same training quality 1.62times sooner than synchronous BF16 training. Code is available at https://github.com/benyucong/train-inference-mismatch-opd