AI看视频的短板：只能记住最后4帧
现在的AI看视频评测，基本是“看几秒片段、选个答案”，这让一个只处理最后4帧的简单模型都能和复杂模型打平——因为选项本身就泄露了答案。这篇论文做了个更狠的测试：让AI看平均88分钟的长视频，然后回答开放式问题，还要能主动互动、调用工具。结果发现，AI在“持续看”和“长期记”之间左右为难：只记最近的，就忘了远的；把过去转成文字，就丢了画面细节；反复压缩记忆，细节就糊了。研究者用“双工人”架构缓解了这个问题：一个工人负责实时响应，另一个在后台慢慢建长期记忆、随时翻旧账。它不是你明天能用上的东西，但它提醒你：AI看视频的“记忆力”，比你以为的差得多。
📄 原文摘要(英文)
Deploying autonomous multimodal agents in continuous, real-world environments requires them to ingest unbounded audio-visual streams and maintain hour-scale memory. However, current evaluations predominantly rely on brief clips and multiple-choice formats. This design allows minimal baselines that process only the last four frames to match or surpass complex streaming models, while answer options also expose language shortcuts. We introduce StreamArena, a benchmark for hour-scale, interactive streaming video understanding. StreamArena contains 243 full-length videos averaging 88.8 minutes and 3,646 rigorously annotated, open-ended question-answer pairs that evaluate real-time perception, historical retrospection, proactive interaction, and multimodal tool utilization. Evaluation across diverse systems exposes a tension between continuous interaction and long-horizon multimodal comprehension. Methods that retain only recent frames cannot recover distant events, methods that convert past observations into text lose visual evidence, and methods that repeatedly compress visual memory struggle to preserve fine-grained details over time. We address this tension with StreamMind, a two-tier architecture that assigns latency-critical interaction and proactive monitoring to independently scheduled frontend workers, while backend workers asynchronously construct persistent multimodal memory and perform historical recall and external search. StreamMind outperforms existing streaming baselines across all four capabilities and reduces query-to-answer latency by reusing persistent state.