看长视频的AI，选哪几帧比看多少帧更重要
长视频AI没法逐帧看：一小时视频每秒抽一帧就是3600张图，系统只留一小撮。以前大家把「留哪几帧」当预处理细节，这篇证明它才是最大的杠杆：在小时级视频上，AI自己挑的8帧，比均匀抽16帧还准6.9分。更意外的是，一个几十年前的稀疏逼近算法(OMP)，没做任何定制，就追平甚至超过了所有专门设计的选帧器。压缩每帧画质几乎不花钱，省下的算力拿去多看一倍帧数，还能再涨两三分——但前提是你真把省下的钱花出去。它不是你明天能用上的，但给所有做长视频AI的人指了个方向：与其堆算力，不如先想清楚该看哪。
📄 原文摘要(英文)
Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time: selection, spatial compression, and reinvestment of the savings, across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.