机器人视频预测:动作差一点,结果差很多
世界模型现在能生成很逼真的视频,但有个要命的毛病:它分不清「几乎一样的动作」和「稍微改一点的动作」会带来什么不同后果。而现实里,机器人拧螺丝时手偏两毫米,可能就从拧上变成滑脱。研究者往训练数据里故意注入时间相关的动作噪声,再要求模型生成的视频对这种噪声的反应和真实录像一致,这样不用反复真机重跑,也能教会模型「动作微变→画面怎么变」。在真实机器人 DROID 上,模型预测的成功率和真机成功率之间的相关性从 0.47 提到 0.79,画面误差也降了最多 23%;拿它来选动作,任务成功率最多涨 8.3%。做机器人策略评估、想让模型先在仿真里试错再上真机的人,这是能省掉大量真机试错成本的那类进展。
📄 原文摘要(英文)
World models aim to predict environment dynamics under actions, enabling the prediction of action consequences without executing them in the real world. Yet despite producing increasingly realistic videos, current world models often fail to capture how small action changes affect future observations. This limits their use in tasks such as robotic policy evaluation and model-based planning, where nearly identical actions can lead to qualitatively different outcomes. To address this, we propose AFFECT, an auxiliary training objective for learning the distribution of visual changes induced by action perturbations. Specifically, we inject temporally correlated noise into the actions of a policy trained on task-agnostic human teleoperation. We then train the world model so that its generated videos respond to this noise as the recorded videos do, without requiring repeated executions from the same initial state. Across OGBench simulation and real-world DROID and ALOHA-2 setups, AFFECT makes world-model rollouts more faithful to real outcomes. On DROID, the Pearson correlation between policy success rates in the world model and on the real robot rises from 0.47 to 0.79, alongside up to 23% lower FVD. These gains carry over to planning, where selecting actions with AFFECT-trained world models improves task success by up to 8.3%.