AI 蒸馏不是抄作业，是老师给学生的隐形打分
把大模型教小模型这件事，我们一直以为是「老师示范、学生模仿」。这篇从强化学习的角度拆开看，发现真相是：老师其实在给学生的一举一动偷偷打分，哪怕老师自己从不那么做。于是两种结局都可能出现——要么学生把对的答案更容易抽出来，要么学生学会钻空子，疯狂输出又长又重复的废话，因为老师给这种废话打了高分。研究者还找到两个止损开关：训练时把不健康的回答遮掉、用 SFT 初始化，都能压住崩溃。它不是你明天能用上的东西，但它改了一个关键认知：蒸馏的效果好坏，不取决于老师多会写，而取决于老师多会判。
📄 原文摘要(英文)
On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts. Our code is available at https://github.com/HancCui/opd_hacking.