人类视频能直接教机器人干活了
机器人训练数据贵得离谱,而人类第一人称视频满地都是——但两者动作对不上,过去的方法只能让模型按「画面长得像」来聚类,学到的不是动作本身。这篇不要求配对数据,而是拿人类片段和机器人片段各自的文字描述去算语义相似度,把相似的人类潜在动作「拉」向对应的机器人潜在动作,同时用机器人自身的动作重建把整个空间钉在控制上。结果是同一套潜在动作信号能同时喂人类视频和机器人视频。更关键的是两条缩放轴都成立:预训练时人类视频越多,下游机器人动作预测误差越低;后训练时任务相关的人类视频越多,真机任务成功率越高。做机器人训练、想省采集成本的人,这是能直接改数据配方的东西。
📄 原文摘要(英文)
Embodied foundation models hinge on diverse robot interaction data, which are costly to collect. Egocentric human videos offer a complementary source of experience, and latent action models (LAMs) provide a popular means of leveraging these videos for policy supervision. Compared with world-action models (WAMs), LAMs can offer greater training efficiency and potentially lower inference costs by avoiding explicit pixel prediction during policy learning and execution. However, most existing LAMs learn latent actions solely through video reconstruction, which can cause their representations to cluster by visual appearance rather than by the underlying actions. As a result, they struggle to align human actions with their robot counterparts. To bridge the gap, we introduce SALA, which emphasizes establishing the semantic correspondence for latent actions. Without requiring paired data, we use the language descriptions of unpaired human and robot clips to estimate semantic similarities and supervise cross-embodiment correspondence. Specifically, we translate these similarities into soft neighborhood structures that guide human latent actions toward semantically related robot latents, while robot action reconstruction grounds the latent space in control. The resulting shared latent action space provides a unified supervision signal for training an embodied foundation model on both human and robot videos. Notably, latent action supervision enables scaling along two axes: (1) pre-training on increasing amounts of general human video consistently reduces robot action prediction error on downstream tasks, and (2) post-training on increasing amounts of task-specific human video substantially improves real-world robot task success rates.