让AI用语言想象未来，训练成本省6倍
现在的AI世界模型各学各的：机器人数据训一个、游戏数据训一个、视频数据训一个，互不相通。这篇把它们的“共同语言”统一成自然语言——不管是机器人转了几度、还是画面里物体移动了，都用一句话描述，然后让同一个模型同时学所有数据。结果：规划效果不输甚至超过各领域单独训练的模型，训练成本省6倍、规划速度快近14倍。它不是你明天能用上的东西，但“用语言统一所有AI经验”这个思路，可能改变未来机器人学新技能的效率。
📄 原文摘要(英文)
This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning. To this end, we introduce LaLa-World, a Language-conditioned Latent World model that learns from simulation and robot trajectories together with broader image and video data. Two designs make this work. First, we describe both numerical actions and visual changes in language, enabling joint training of a single future predictor across heterogeneous datasets. Second, we build the predictor on a language model, allowing it to interpret language conditions while predicting future latent states. The resulting predictor achieves higher average MPC planning performance than existing LWMs trained separately for each domain or task. Adding image and video data further improves average planning performance and robustness to action phrasing. In a matched comparison with an RAEv2-based video generation model baseline, LaLa-World reaches comparable planning performance with 6times less training cost and 13.9times faster planning. Beyond planning, the pretrained LaLa-World serves as an initialization for LaLa-Policy, which jointly predicts actions and future visual representations. LaLa-Policy performs competitively with Cosmos Policy on LIBERO and RoboCasa, with up to 5.5times faster inference.