RL训大模型的两难，被一个「换边」破解了
训练大语言模型时有个经典两难：你想让模型探索新回答，又怕它跑偏，于是加个约束——但约束本身会吃掉探索的预算。阿里团队发现，问题出在约束放错了地方：传统方法约束的是模型「输出什么」(action side)，而他们换成约束「输入什么」(query side)。具体做法是加一个 Query-KL 项，监控训练过程中模型对问题的偏好是否偏离了初始分布，并给每个问题配一个权重，让模型更倾向于回答那些「典型」的问题。关键设计是：这个约束的梯度只影响模型对问题的理解，不影响它对回答的探索——探索预算被完整保留。在六个数学推理基准上，ERPO 替换了标准 Policy-KL，在高温度解码和长周期训练下表现更稳定、准确率更高。它不是你明天就能直接用的工具，但提供了一个思路：当优化陷入两难，换个约束对象可能比加更多约束更有效。
📄 原文摘要(英文)
Policy optimization (PO) for Large Language Models faces a stability--exploration trade-off, currently mediated by an action-side Policy-KL regularizer. This puts practitioners in a double bind: keeping Policy-KL constrains response behavior and consumes the action-side exploration budget, while dropping it leaves the optimization without an explicit drift control. We argue for an alternative that breaks the dilemma by moving regularization to the input side. As training progresses, the distribution over training queries induced by the current policy drifts unchecked from its pre-RL reference distribution. Concretely, Environment-Regularized Policy Optimization (ERPO) introduces a Query-KL (QKL) term that bounds this query distribution shift, together with a dataset-static reference-derived per-query weight that biases each per-query update toward queries typical under the reference. The QKL gradient flows strictly through the query likelihood; the response score function used by policy-gradient estimators does not appear in the QKL term, so QKL exerts no direct gradient pressure on the response distribution---exploration is preserved. ERPO plugs into GRPO/PPO/REINFORCE-style pipelines without additional forward passes. On six mathematical reasoning benchmarks, ERPO replaces the standard Policy-KL regularizer while achieving effective control over query distribution drift, delivering stronger accuracy and substantially more stable behavior under high-temperature decoding and long-horizon training.Our source code are available at https://github.com/alibaba/ERPO