AI 蒸馏的「双刃剑」:教推理还是教答案？
大模型「蒸馏」——用强模型教弱模型——有个常见直觉：学生学的是老师对具体问题的答案。这篇论文用控制实验告诉你，直觉是错的。当学生用自己的思路生成轨迹、老师再纠正(即 on-policy distillation)，学生真正学到的是老师的推理方式，而不是答案。证据是：训练时题目难不难几乎不影响效果；甚至老师自己都解不出的题，学生也能从中受益。但问题来了：这种「推理迁移」的广度取决于老师和学生是否「同源」——同源组合下，学生能在不同语言、不同推理长度甚至不同领域里接近老师；跨源组合则基本只拟合训练分布。这意味着，当你把多个领域专家模型蒸馏进一个学生时，你没法让每个专家的影响只局限在自己的领域——它们会互相拉扯，像跷跷板一样此消彼长。这不是你明天能用上的技巧，但它解释了一个常见困惑：为什么多专家蒸馏常常不如单专家蒸馏稳定。
📄 原文摘要(英文)
On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly understood, as most studies evaluate OPD on a single domain and on benchmarks close to the training data. We present a controlled study that varies one generalization factor at a time, from in-domain distribution shifts to cross-domain transfer and the multi-teacher setting. We find that OPD transfers a teacher's reasoning behavior rather than its answers to particular problems: training difficulty barely matters, and even problems the teacher never solves are useful. Transfer depends strongly on the origin relationship between teacher and student: same-origin pairs bring the student close to the teacher across languages, reasoning horizons, and even other domains, whereas cross-origin pairs mostly fit the trained distribution. This broad reach is a double-edged sword: since routing prompts to domain experts cannot confine each teacher's influence, combining them yields a mixture-dependent seesaw among their capabilities. These results clarify when OPD generalizes and offer a useful perspective for diagnosing multi-teacher OPD.