AI训练提速3.8倍，但数据过期怎么办？
大模型训练有个瓶颈：生成数据（rollout）和更新模型必须同步，像流水线上一环卡一环。这篇论文把两者拆开，让生成和更新异步跑，训练速度提升1.6到3.8倍。但代价是模型更新时用的数据可能已经“过期”——基于旧版本模型生成的。他们发现，不同训练方法对过期数据的容忍度不同：一种叫“前向KL”的方法更抗过期，而“反向KL”则很脆弱。对于脆弱的反向KL，他们试了各种稳定异步训练的招数，结果都不如一个简单办法：在更新时用当前模型重新计算损失信号。他们还分析了有限缓存带来的偏差-方差权衡，提出用多采样蒙特卡洛来降低方差。最终开源了AsyncOPD框架。这不是你明天能直接用的工具，但它揭示了异步训练中一个关键权衡：速度提升与数据新鲜度之间的取舍，对做大模型训练的人有参考价值。
📄 原文摘要(英文)
On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training. Like reinforcement learning (RL), however, OPD faces an on-policy systems bottleneck, as rollouts can dominate training time for reasoning workloads. Asynchronous training pipelines can alleviate this bottleneck by decoupling rollout generation from learner updates, but doing so introduces stale-policy data. While prior work has studied stale data in asynchronous RL, its effects in OPD remain underexplored. We present the first systematic study of staleness in asynchronous OPD, focusing on a practical setting where teacher feedback is implemented through local KL losses and full-vocabulary teacher logits are too expensive to store or transfer, necessitating finite teacher-score caches. We first show that KL direction changes the stale-data problem: teacher-weighted forward KL is more robust to stale rollouts, whereas student-weighted reverse KL is vulnerable. Second, for this vulnerable reverse-KL case, we study whether methods designed to stabilize asynchronous RL can mitigate OPD staleness. In our experiments, they do not improve over a simpler OPD-specific surrogate: recomputing the reverse-KL signal under the current student at learner time. Third, we analyze how finite teacher-score caches create a bias-variance tradeoff for sparse and sampled reverse-KL OPD estimators. This motivates multi-sample Monte Carlo (MC), which preserves MC correctability while reducing one-sample variance. Finally, we present and open-source AsyncOPD, a fully asynchronous OPD training pipeline built from these estimator choices. Experiments show that AsyncOPD improves training throughput by 1.6times to 3.8times over strict synchronous training while reaching comparable accuracy.