大模型蒸馏，别急着用在线数据
训练小模型时，主流做法是让老师实时生成数据来教学生，叫「在线蒸馏」。但一篇覆盖 17 组师生模型(最大 2350 亿参数)的研究发现：用学生自己一开始生成的旧数据来教，反而在 14 组里更准，最高提升 13.6%，训练还快 11.4 倍。原因在于，只有当老师和学生「思路高度一致」时，在线数据才有优势；否则学生越学越偏，老师给的数据反而成了噪音。对做模型压缩、部署小模型的人来说，这提醒你：别默认在线蒸馏更好，先算一下师生输出重合度，再决定数据从哪来。
📄 原文摘要(英文)
On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.