小模型练出大模型推理，省下80%训练成本
训练大模型推理能力通常需要它自己反复试错（RL），成本极高。这篇论文发现：让小模型先做RL，然后把它学会的“偏好变化”当作奖励信号传给大模型，大模型不用自己跑RL就能获得类似提升。具体做法是拿小模型RL前后的两个版本做对比，把它们的输出概率差作为隐式奖励，在大模型自己的输出上微调。实验显示，Qwen3-1.7B在AIME数学题上从48.3%跳到58.3%，只用了8张A100跑4小时，而直接在大模型上跑RL需要更久。这不是你明天能直接用的工具，但它指向一个趋势：未来的模型训练可能不再每代都从头跑RL，而是靠小模型“探路”、大模型“抄近道”。
📄 原文摘要(英文)
Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.