AI 终于能边看边听你说话，而不是先转成文字
现在的 AI 助手跟你对话，得先把你的话转成文字、把视频抽成几帧，再拼起来理解。这篇把「看视频+听声音」直接做成 AI 的原生能力：你对着镜头说话，它同时接收画面和声音，你的表情、周围的环境、手里的东西都是它理解你的一部分，不用先转译。研究者还造了一套多智能体系统，自动生成大量这种「边看边聊」的训练数据，并设计了一套奖励机制，让 AI 在回答正确的同时，也学会注意效率(别问太多废话)和风格(像真人聊天)。用这套方法训练的模型，在机器生成的测试和真人录制的测试上都明显变强。它不是你明天就能用上的产品，但这是 AI 从「听懂你」走向「看懂你」的关键一步。
📄 原文摘要(英文)
We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.