小模型练出大模型推理能力，省掉80%训练成本
训练大模型推理能力通常需要它自己反复试错（RL），成本极高。这篇论文发现：让小模型先做RL，然后把小模型“学会的偏好”作为信号直接喂给大模型，大模型不用自己试错就能获得同样的推理提升。具体做法是拿小模型RL前后的两个版本做对比，把差异当成“隐式奖励”来指导大模型。在AIME数学竞赛上，1.7B的小模型帮1.7B的大模型（同规模但更强）把正确率从48.3%提到58.3%，只用8张A100跑4小时——同等效果直接RL要贵得多。这不是你明天能用的工具，但它指向一个趋势：未来训练大模型可能不再需要每次都从头跑RL，小模型的经验可以像“知识”一样迁移。
📄 原文摘要(英文)
Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.