AI 开口说话时，终于有了脸
现在的 AI 助手能听、能说、能看，但回答时你只听见声音，看不见它。这篇让 AI 在说话时同步生成一张会动的脸：它先在心里写一份「视觉剧本」——场景、情绪、动作——再照着剧本把语音和画面逐帧对齐，嘴型、表情和语气是同一个节奏。关键是它不需要海量的「问题-回答-语音-视频」配对数据，而是把语音和画面当成同一根时间轴上的两条轨道，分别从不同数据里学，再拼起来。生成视频还做了提速，四步推理就能跑完，四块 GPU 下延迟接近实时。它不是你明天就能用的产品，但这是「AI 有身体」这条路上一个具体的工程节点。
📄 原文摘要(英文)
Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, yet their responses remain visually disembodied. We introduce Ex-Omni-2D, an omni-modal dialogue framework that generates a coordinated response comprising text, personalized speech, and reference-conditioned video. Given a multimodal query, reference image, and reference audio, the model predicts a structured Visual Thought Plan (VTP) describing scene, emotion, and motion, followed by response text and native multi-codebook speech units. These units form a shared acoustic-temporal interface: they are decoded into speech and aligned online with video frames. This interface enables the response and avatar pathways to be learned from heterogeneous speech, dialogue, and avatar-video data, avoiding the need for large-scale query--text--speech--video supervision. A full-sequence Video Generator serves as the primary Teacher. For efficient incremental generation, we further distill it into a few-step block-causal Streaming Student whose Prefix Streaming mechanism carries a clean latent across consecutive chunks to reduce cumulative late-chunk degradation. With four-step inference, the complete four-GPU pipeline achieves an end-to-end RTF of 1.293 at 400times720/720times400, providing a practical quality--efficiency operating point.