异步强化学习：用“过时”数据训练AI反而更稳
异步强化学习能大幅提升训练吞吐，但一个核心矛盾是：模型在训练时用的数据可能已经“过时”（stale），导致训练和推理不一致，甚至让模型崩溃。传统PPO算法只对采样到的更新做裁剪，无法有效控制高过时度的更新。这篇论文提出了一种“过时自适应信任域”（SAT）方法：它把过时程度作为信号，对过时严重的更新施加更保守的约束，而对正常数据保持原有行为。在Qwen3-30B模型上测试，SAT在AIME24评测中达到35.83（滞后1步）和34.79（滞后8步），显著优于基线。这不是你明天能直接用的技术，但它揭示了异步训练中一个被忽视的稳定化杠杆——不是拒绝过时数据，而是根据过时程度动态调整更新幅度。
📄 原文摘要(英文)
Asynchronous reinforcement learning improves throughput by decoupling rollout generation from optimization, but staleness is an inevitable byproduct compounded by policy lag, engine delays, and mixture-of-experts routing. From a trust-region perspective, this mismatch is critical: training-inference divergence governs approximation error in finite-horizon bounds, whereas PPO clipping only gates sampled outward updates, acting as a sampled surrogate rather than a full-policy constraint. As a result, high-staleness updates remain weakly controlled in the asynchronous regime where stale rollouts matter most. We introduce the Staleness-Adaptive Trust Region (SAT), which uses the detached sampled log-ratio as a practical staleness proxy, identifies high-mismatch tails within each batch via staleness-based kernel scaling, and contracts only the sign-selected endpoint of the nominal PPO interval. This preserves baseline behavior on ordinary tokens while enforcing more conservative updates on newly intercepted outward bands. We prove local interval containment and pointwise pessimism relative to PPO, showing how the adaptive rule reshapes update geometry under heterogeneous staleness. We evaluate SAT in a decoupled asynchronous RL setup built on Qwen3-30B-A3B-Base, using SGLang as the inference engine and Megatron for training. In this setting, SAT-GSPO w/ R3 achieves the best observed AIME24 avg@8, reaching 35.83 at lag 1 and 34.79 at lag 8, while SAT-GSPO reaches 34.17 at lag 1. Adaptive clipping and routing replay act as complementary stabilizers targeting mismatch tails and routing inconsistency, respectively. Overall, aligning clip intervals with staleness heterogeneity effectively stabilizes asynchronous RL.