用语言指令预测物体3D运动轨迹
你对着视频说一句「把杯子推到左边」，AI就能预测出杯子上每个点接下来会怎么移动——不是2D画面里的像素，而是真实世界中的3D坐标。研究者先让模型看几帧视频，再给它几个3D点（比如杯口和杯底），加上你的语言指令，它就能输出这些点未来的3D轨迹。为了训练这个能力，他们从116万个无标注视频中自动生成了100万条带语言描述的3D点轨迹数据，并人工验证了一个覆盖111类物体、61种运动类型的评测集。模型在评测集上大幅超越现有方法，而且学到的3D运动先验能直接迁移到机器人操作（提升训练效率和泛化性）和视频生成（让AI生成的视频里物体动得更真实）。它不是你明天就能用的产品，但指向了一个方向：让AI理解物体在三维空间中的运动，并听从语言指令。
📄 原文摘要(英文)
Motion forecasting is central to visual intelligence: agents must anticipate how objects will move in order to plan actions, reason about physical interactions, and synthesize realistic futures. We argue that 3D points in world coordinates provide a general representation that is class-agnostic, view-stable, compact, and directly useful for downstream tasks. We formalize the task of goal-conditioned 3D point motion forecasting: given a short visual history, a set of 3D query points on an object of interest, and a language description of the intended goal, the model predicts the future 3D trajectory of each point. We introduce a full stack to study this task at scale: (1) MolmoMotion-1M is a large corpus of action-described, object-grounded 3D point trajectories annotated from 1.16M unconstrained videos; (2) PointMotionBench is a human-verified benchmark spanning 111 object categories and 61 motion types; and (3) MolmoMotion is a general motion forecasting model that supports both autoregressive coordinate prediction and flow-matching-based trajectory generation. MolmoMotion accurately predicts diverse motion patterns with different language instructions, and significantly outperforms existing motion prediction baselines on PointMotionBench. Finally, we show that the learned 3D motion prior transfers well to downstream applications: it improves training efficiency and generalization for robot manipulation, and its predicted trajectories provide effective motion guidance for generative models to synthesize videos with more realistic object motion.