视频理解评测：一半题目不看也能答对
你看到的视频理解评测，可能有一半题目根本不需要看视频。研究者用一套诊断工具检查了现有视频基准，发现55%的样本不依赖视觉或时间信息就能答对——比如靠常识猜、靠字幕蒙。去掉这些捷径后，最强模型的表现只比随机猜好一点点。这不是又一篇刷榜论文，而是告诉你：当前视频AI的真实能力被严重高估了。它不是你明天能用上的，但下次看到“AI看懂视频”的新闻，你可以多问一句：它真的看了吗？
📄 原文摘要(英文)
The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55\% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.