AI 训练机器人翻盘：从 20% 成功率跳到 96%
机器人抓取动作的成功率能从 19.8% 飙到 96.4%，不是换了个更聪明的模型，而是意识到：以前用来防止 AI 学崩的“安全阀”，在数据多得用不完的时候反而成了拖累。研究者发现，传统 off-policy 强化学习里的参数归一化和双 Q 网络这些稳定器，只在数据少时有用；一旦用上大规模并行仿真、数据管够，它们反而限制 AI 学得更准。于是他们按数据量切换配置：数据少用保守版，数据多用激进版，顺手把成功率拉高了近 5 倍。这不是你明天能用上的技术，但它点破了一个趋势：AI 训练的思路正在从“小心别崩”转向“大胆喂数据”，仿真越跑越快，机器人学动作的瓶颈越来越不在算法，而在你怎么配数据。
📄 原文摘要(英文)
Massively parallel simulation changes the data regime in which off-policy reinforcement learning (RL) is trained, challenging stabilizers designed for data-limited replay. Through controlled experiments across eight benchmark families, we show that these stabilizers are data-regime-dependent: parameter normalization helps with narrow replay coverage but restricts value fitting when data are abundant, while clipped double-Q can be relaxed in high-throughput manipulation. Age-biased replay weighting improves learning efficiency across regimes, especially with limited network capacity. Based on these findings, we propose WarpSAC, a regime-aware family of off-policy RL algorithms. WarpSAC uses Sample Weight Decay for efficient exploitation and provides two variants: WarpSAC-L (Norm ON, clipped double-Q) for data-limited CPU-scale training, and WarpSAC-A (Norm OFF, single-Q) for data-abundant GPU-parallel training. WarpSAC improves normalized score--step AUC over FlashSAC by 4.5% across nine CPU-scale environments and 23.1% across fourteen GPU-parallel environments. It increases UnitreeG1TransportBox-v1 success rate from 19.8% to 96.4%, improves mean normalized wall-time AUC on MuJoCo Playground by 19.1%, and achieves 36.4% faster sim-to-real deployment on Unitree G1 than FlashSAC. These results show that scalable off-policy RL should adapt its stabilizers to the available data regime.