大模型训练时表现好，上线却变笨？
大模型用强化学习微调时，经常出现训练时效果不错、上线后却变差的情况。这篇论文发现，问题出在训练和推理用的是两套引擎——训练时为了精度用高精度计算，推理时为了速度用低精度，导致同一个模型在两边算出的概率不一致，相当于训练时学的策略和实际用的策略是两回事。研究者提出一个新框架，在训练过程中加一道检查：只有那些在推理引擎里也能稳定提升效果的更新才被采纳。实验表明，这种方法能显著提升推理任务的稳定性和平均表现。它不是你明天就能直接用的工具，但解释了为什么大模型微调容易翻车，并给出了一个靠谱的解决方向。
📄 原文摘要(英文)
Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse. One vital cause is training-inference mismatch: LLM adopts separate inference and training engines for generation efficiency and training precision, which in practice exhibits inconsistent probabilities for the same trajectories on training and inference sides, even with synchronized model parameters. This naturally induces a special type of off-policyness ever existing and poisoning the training. Prior works have made various efforts in addressing the off-policyness to stabilize the training policies under the mismatch. In this paper, we point out the objective misalignment neglected by existing works that an effective update to the policy in the training engine not necessarily ensures the improvement of the inference policy, i.e., the one used in deployment. To this end, we propose a new policy optimization objective for LLM RL, named Monotonic Inference Policy Improvement (MIPI). Following this principle, we introduce Monotonic Inference Policy Update (MIPU), a two-step LLM RL framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates using an inference-side gap proxy. Experiments conducted on two model scales under high mismatch show that MIPU improves average reasoning performance and training stability.