AI看视频不再逐字想,直接给答案
现在的视频AI看一段画面,得先“想”一遍再回答,慢且费算力。这篇把“看”和“答”合并成一个动作:模型不再逐字生成答案,而是直接对视频片段和候选答案打分,谁分高谁就是结果。在三个时间定位基准上,4B和35B模型比原版分别提升6.7和10.8个点,推理速度快2.8到3.1倍;在四个视频问答基准上,速度提升超过100倍,准确率还和原版相当。它不是你明天就能用的产品,但这是视频AI从“慢思考”走向“快反应”的一个明确信号——以后机器人看监控、看直播,能边看边决策,而不是看完再想。
📄 原文摘要(英文)
Visual agents must turn video observations into timely decisions to track progress and choose what to do next. Building on the emerging System One paradigm, we introduce VideoJev, a family of models that cast video understanding and interaction as structured decision tasks. We adapt pretrained multimodal large language models (MLLMs) by replacing autoregressive answer decoding with a shared head that scores temporal intervals and textual options against video evidence. These scores yield event spans or answer choices, unifying temporal grounding and multiple-choice video question answering. This combines the semantic flexibility of MLLMs with efficient decisions that agents can use directly. As an interaction unfolds, new goals and intermediate results prompt further queries about the same visual history. VideoJev therefore reuses visual computation through Cached inference for queries as they arrive and Joint inference for multiple queries in a single forward pass. Across three temporal grounding benchmarks, VideoJev-4B and VideoJev-35B-A3B improve mean mIoU over their backbones by 6.7 and 10.8 points, respectively, with measured inference speedups of 2.8-3.1times. On four video QA benchmarks, both achieve performance comparable to their backbones in Thinking mode, with measured inference speedups exceeding 100times. These results offer a unified, reusable decision interface for composing structured visual judgments into complex, open-ended workflows.