AI Pulse
📄 论文解读

机器人看视频自学动作

机器人要学会干活,通常得靠人一遍遍示范、还要记录关节角度这类内部数据。这篇让机器人只看普通视频,就能把「手往哪伸、什么时候夹紧」反推出来,再拿同一套三维场景理解去生成自己未来的动作。更关键的是,它能把别人家机器人的视频「翻译」成自己的示范:动作重定向过去、画面也合成出来,新机型只要十段示范就能上手,成功率比只用合成数据训练高出 41.5%。做机器人训练数据、想让一台机器学另一台机器经验的人,这是能省掉大量真机示范的路子。

📄 原文摘要(英文)

Robot manipulation videos provide a rich visual experience, yet recovering robot action and gripper-state annotations from RGB observations still remains challenging. Using the embodiment-agnostic tool-center-point (TCP) pose as the action space, we present TCP4D, a unified model in which a shared 4D scene representation both tracks observed TCP motion and generates future TCP trajectories. Given a monocular video and initial 2D TCP queries, the tracker directly recovers metric TCP poses in SE(3) and gripper states alongside scene geometry and camera parameters. The generator samples local features from the same 4D representation along the recovered TCP tracks, then combines them with robot observations and language instructions to predict future TCP states. Our new cross-embodiment augmentation pipeline converts existing RGB videos into demonstrations for alternative embodiments by retargeting the recovered motion and synthesizing the corresponding observations, without recorded source-robot states. Experiments show that the tracker achieves absolute trajectory errors of 7.60 and 16.07 mm on DROID and RoboLab and adapts to a new embodiment with only ten demonstrations. The generator achieves 69.19% mean success on RoboTwin and outperforms well-established policy models on two real-world tasks, while joint training on source and augmented demonstrations improves target-robot success by 41.5% over augmented-only training.

arXiv 原文

订阅 AI Pulse

每天 08:00 · 12:30 · 18:30 · 23:50 更新