AI 蒸馏的秘密：老师其实没在教
主流观点认为，让大模型向更强的老师学习，靠的是老师逐字逐句的指导。这篇论文把老师的指导拆开看，发现大部分指导其实是噪音——而且老师越强，噪音越多。更意外的是，学生根本不在乎这些噪音，删掉它们成绩不变。真正起作用的，是学生自己在压低那些它本来就不太会选的词的概率，这根本不需要老师。研究者据此做了一个去掉老师、只靠学生自身熵值调节的训练方法，在数学竞赛题上比原模型提升 263%，还超过了传统蒸馏。它不是你明天能用上的，但它动摇了『蒸馏=向老师学』这个直觉，指向一个更省事的训练方向。
📄 原文摘要(英文)
On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.