AI 终于能边看边听你说话，而不是先转成文字
现在的多模态 AI 跟你对话，得先把你的话转成文字、把视频抽成几帧，再拼起来理解。这篇把流程砍了：模型直接同时接收你的声音和画面，你的问题就藏在音视频里，没有中间的文字步骤。研究者造了个多智能体引擎，自动生成大量「真人边说话边比划」的对话数据来训练和评测，还设计了一套奖励机制，同时优化回答对不对、快不快、像不像人。训练后的模型在真人录制的对话上表现也变好了。它不是你明天能用上的东西，但「AI 看着你的表情、听着你的语气直接回应」，是下一代交互的方向。
📄 原文摘要(英文)
We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.