把视频模型变成万能动作翻译器
同一个动作，人和机器人做出来画面完全不同，但本质是同一件事。现有模型学动作时总把「长相」和「动作」搅在一起，换个身体就不认了。DreamX 把预训练好的视频扩散模型改造成动作提取器：它把每帧视频压缩 300 倍，只留下「动作」本身，丢掉长相、环境这些干扰。结果是人做的动作，机器人能直接照做，零样本跨身体、跨环境迁移，还能在压缩后的动作空间里训练策略，省下大量算力。这不是你明天能用的工具，但它指向一个方向：动作本身可以成为一种通用语言，人和机器共享。
📄 原文摘要(英文)
Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by 300times (1{,200 rightarrow 4} tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.