AI看视频学做事，关键帧是命门
现在的AI看视频答题已经不错了，但让它看完教程后动手操作——比如跟着烹饪视频做菜、照着软件教程点按钮——几乎全翻车。研究者发现，问题出在AI不会挑关键帧：它把每一帧都当重点，结果淹死在无关画面里。他们搞了个新评测集，专门考AI「看完视频能不能干活」，同时发明了一个挑帧算法，既看画面跟任务有没有关，又看场景有没有变化。在标准视频问答测试上，这个算法把准确率提升了2个百分点；在干活任务上，效果更明显。它不是你明天能用上的，但点出了一个方向：AI要真学会做事，得先学会「什么时候该看哪一眼」。
📄 原文摘要(英文)
Video understanding is a fundamental capability for multimodal intelligence, and recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance on Video Question Answering (VideoQA) benchmarks. However, existing benchmarks primarily evaluate whether models can perceive shallow visual cues, while rarely examining whether MLLMs can learn deeper knowledge or procedural skills from video tutorials and generalize them to downstream long-horizon agentic tasks. To address this gap, we introduce VG-GUIBench (Video-Guided GUI Benchmark), a new benchmark designed to evaluate whether MLLM-based GUI agents can follow video tutorials to complete corresponding GUI interactive tasks. Furthermore, we observe that the performance of models on both VideoQA and video-guided agentic tasks critically depends on effective keyframe extraction. Based on this observation, we propose TASKER (Task-driven And Scene-aware Keyframe searchER), a keyframe extraction algorithm that jointly considers task relevance and scene dynamics to identify informative frames. Experimental results demonstrate that TASKER achieves significant performance improvements on both VideoQA and video-guided agentic task benchmarks, outperforming the best baseline by 2.0% on the EgoSchema fullset and 1.8% on the NExT-QA dataset, respectively. These results further highlight the potential of generalized keyframe extraction methods for video understanding tasks. Our code and data are available at https://github.com/VG-GUI-TASKER/VG-GUI-TASKER.