AI Pulse
📄 论文解读

AI看长视频不再从头看到尾,而是像侦探一样主动找线索

现在的AI看长视频,无论内容多无聊,都得一帧一帧看完,耗时耗力。这篇论文让AI像侦探一样:先扫一眼,然后根据问题主动去“听”或“看”关键片段,把信息记在脑子里,再决定下一步。它用了一个循环——观察、思考、行动——每次只处理最相关的部分。结果,一个7B参数的模型在LVBench上超过了10倍大的Qwen2.5-VL-72B(50.5% vs 47.3%)。这不是你明天能用的工具,但它指明了一个方向:未来的AI会更聪明地分配注意力,而不是傻傻地看完所有数据。

📄 原文摘要(英文)

Passive models for long video understanding typically rely on a "watch-it-all" paradigm, processing frames uniformly regardless of query difficulty, causing computational cost to grow with video duration. Although interactive frameworks have emerged, they often rely on global pre-scanning, and their context cost still scales with video length. We propose OmniAgent, the first native omni-modal agent that formulates video understanding as a POMDP-based iterative Observation-Thought-Action cycle. OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory, effectively decoupling reasoning complexity from raw video duration. To operationalize this, we introduce (1) Agentic Supervised Fine-Tuning to bootstrap native active perception via best-of-N trajectory synthesis with dual-stage quality control, and (2) Agentic Reinforcement Learning with TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage), which leverages turn-level entropy to steer credit assignment toward pivotal discovery turns. Crucially, OmniAgent exhibits positive test-time scaling, where performance improves as the number of reasoning turns increases, validating the efficacy of active perception. Empirical results across ten benchmarks (e.g., VideoMME, LVBench) demonstrate that OmniAgent achieves state-of-the-art performance among open-source models. Notably, on LVBench, our 7B agent outperforms the 10times larger Qwen2.5-VL-72B (50.5% vs. 47.3%).

arXiv 原文

📬 订阅 AI Pulse

每天三次更新,不错过重要信号

▲ 回到顶部