机器人学动作,不看说明书,改看120万小时人类视频
机器人学抓东西,通常得靠人类手把手标注动作数据,又贵又少。这篇直接换了个思路:让机器人先看海量第一人称视频——人怎么拿杯子、用工具、开抽屉——把物理世界的规律先学进脑子里,再用少量机器人数据微调,最后部署时用一套“慢思考+快反应”的双系统架构,在普通显卡上就能以每秒30次的频率实时预测动作。结果很硬:只靠自家机器人数据时成功率36.1%,加上12万小时视频预训练后,零样本直接跳到77.8%,没见过的任务提升尤其明显。它不是你明天就能装进自家机械臂的东西,但方向很明确:机器人不再需要被逐条编程,而是像人一样,靠“看别人怎么做”来学会做事。
📄 原文摘要(英文)
Robotic manipulation faces a fundamental scaling challenge: robust generalization demands broad physical experience, yet action-labeled robot trajectories are expensive to collect and inherently limited in diversity. Egocentric videos offer a far more scalable source of embodied experience, capturing object interactions, contact dynamics, tool use, and long-horizon behaviors across diverse environments. The central challenge is how to convert this abundant but action-free experience into effective robot control. We introduce ZimaBlue, a scalable framework for learning generalizable World Action Models (WAMs) from large-scale video. ZimaBlue follows a three-stage training curriculum: it first performs causal embodied video pre-training on large-scale human and robot egocentric videos, then grounds the learned visual dynamics in heterogeneous robot trajectories through video-action mid-training with a unified action representation, and finally specializes the model to a target robot for deployment. To make generative WAMs practical for real-time control, ZimaBluefurther adopts an asynchronous Slow-Fast dual-system architecture, where a high-capacity Slow world model provides generalizable spatiotemporal representations and a lightweight Fast branch enables 30 Hz action prediction on NVIDIA RTX 4090. On real-robot zero-shot evaluations, scaling from target-robot data alone to over 120,000 hours of embodied video improves success from 36.1% to 77.8%. ZimaBlue further delivers strong performance across multiple benchmarks, with particularly pronounced gains on unseen tasks.