AI教学暴论：老师不重要，把差的答案压下去就行
都说蒸馏是学生跟老师学；这篇发现，老师的批改大部分是噪音，学生照样能进步。拆开看，收益主要来自把模型自己生成时概率极低的残次 token 压掉，跟老师给不给分关系不大。顺着这个发现，研究者干脆去掉老师，只按位置的混乱程度自适应打压低概率 token、把概率匀给高概率候选，Qwen3-1.7B 在 AIME24 上平均分涨了 35.41 分。这不是说老师没用，而是指了个更省事的方向：与其费劲跟好老师学，不如自己把明显错的路堵死。想给推理模型做低成本后训练、又不想依赖强教师模型的人，这是眼前就值得试的东西。
📄 原文摘要(英文)
On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.