AI 看视频,学会先开口
现在的 AI 助手都是你问它才答;这篇让它反过来——盯着视频流,自己判断哪一刻值得说话,然后主动开口。AURA 2.0 做三件事:在画面变化里挑出值得聊的话题、在合适的时机不等提问就发起对话;看到东西能自己调外部工具,把查回来的信息接进当前对话;还能在画面上框出你说的或它提到的物体,让回答落到具体的人和区域。为了撑住长时间视频,它带了一层跨长视频的记忆,让上下文不断线。公开流式评测里它在开源模型里领先。它不是你明天能装进手机的东西,但方向很清楚:AI 从「等你问」变成「替你先看见」。
📄 原文摘要(英文)
Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) lets the model act on what it sees by invoking external tools using the current visual and conversational context and incorporating returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities.