AI 开口说话时，终于有脸了
现在的 AI 助手能听、能说、能看，但回答时你只听见声音，看不见它。这篇让 AI 在说话的同时生成一张会动的脸：它先在心里写一份「视觉剧本」——场景、情绪、动作——再照着剧本同步输出语音和口型、表情都跟得上的视频。关键是它把语音和视频共用同一套时间轴，所以不用海量「问题-回答-语音-视频」配对数据，用现成的语音对话和视频素材就能训练。生成速度也够用：四张显卡、四步推理，720p 视频接近实时。它不是你明天就能装进手机的功能，但这是「AI 有实体感」这条路上少见的完整一步。
📄 原文摘要(英文)
Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, yet their responses remain visually disembodied. We introduce Ex-Omni-2D, an omni-modal dialogue framework that generates a coordinated response comprising text, personalized speech, and reference-conditioned video. Given a multimodal query, reference image, and reference audio, the model predicts a structured Visual Thought Plan (VTP) describing scene, emotion, and motion, followed by response text and native multi-codebook speech units. These units form a shared acoustic-temporal interface: they are decoded into speech and aligned online with video frames. This interface enables the response and avatar pathways to be learned from heterogeneous speech, dialogue, and avatar-video data, avoiding the need for large-scale query--text--speech--video supervision. A full-sequence Video Generator serves as the primary Teacher. For efficient incremental generation, we further distill it into a few-step block-causal Streaming Student whose Prefix Streaming mechanism carries a clean latent across consecutive chunks to reduce cumulative late-chunk degradation. With four-step inference, the complete four-GPU pipeline achieves an end-to-end RTF of 1.293 at 400times720/720times400, providing a practical quality--efficiency operating point.