用更弱的AI教更强的AI，反而教得更好
训练大模型有个死结：想让它更强，得有个更强的老师，可最强的模型已经是天花板了。这篇论文把思路倒过来——用几个更弱的模型当老师，反而能教出更强的学生。做法是拿一对弱模型（比如同一个模型强化学习前和后，或者大一点和小一点的底座）做减法，把它们的差异当成“能力方向”，加到学生自己身上，合成一个虚拟老师，再让学生照着学。在数学和代码题上，学生不仅超过了老师，而且就算所有监督源都比它弱，它还能继续变强。这暗示“变强”不一定要靠更强的外部指导，有时把已有能力的差异提炼出来，自己就能往上走。
📄 原文摘要(英文)
On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs. Prevailing approaches assume a teacher at least as capable as the student: they either distill a larger model into a smaller one, which fails at the frontier where no larger teacher exists, or consolidate multiple domain experts trained from a shared base, which requires costly training at the student's scale. We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a simple yet effective OPD framework that improves the strong student by distilling from multiple weak models. W2S-OPD constructs a proxy teacher in logit space from a contrast pair of a positive and a negative model, both smaller than the student and cheap to obtain. Their logit difference isolates the capability direction, which is added to the student's own base model, yielding a proxy teacher that couples this direction while staying distributionally adjacent to the student. The student then distills it by minimizing the per-token reverse KL on its own rollouts. We instantiate the contrast pair as i) a post-RL expert against its pre-RL initialization, isolating the skill RL instills, ii) a larger against a smaller base model, isolating the capability from scale, and iii) a small base model with correct versus wrong hints, isolating the instance-level direction toward the solution. Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker. Analysis shows different contrasts yield distinct signals: the post-RL and hint contrasts emphasize reasoning frameworks, while the scale contrast emphasizes the solving procedure. Our code will be available at https://github.com/Yu-Fangxu/W2S-OPD.