AI Pulse
📄 论文解读

AI看视频,其实没在看「动」

视频模型能答对问题,却答不出「这段画面里东西动得多快」——同一批素材,直接从画面特征里读速度能到 87%,接上语言模型后却掉到瞎猜水平。原因是训练时每句描述、每个问题只提到画面的一两个属性,模型于是学会了「按关键词去检索」,而不是真的去看视频;编码器里本来就存着的运动信息,就这么被晾在一边。研究者发现这类预测式编码器把「怎么动」存在一个跨视频共用的子空间里,而逐帧对齐文字的编码器(如 SigLIP)没有这个共用成分。他们用像素变换把公开视频改造成答案由变换决定、只在「变化」上不同的成对样本,逼语言模型去读那个子空间。单阶段、453 万样本、每帧 32 个视觉 token 训出的 V-JEPA-LM(1B 到 8B),在十一个视频基准上各尺寸档平均分都领先;4B 版拿 72.1,最好的开源 4B 是 61.1、最好的开源 8B 是 64.8。对照实验把功劳分得很清:编码器决定「能读到什么」,监督方式决定「实际读了什么」——同等预算下,反事实监督平均加 4.5 分,自然基准里的速度题加 38 分。它不是你明天能用上的东西,但它说明现在多数视频 AI 的短板不在看得不够多,而在没人教它看「变化」。

📄 原文摘要(英文)

World-model video encoders such as V-JEPA learn how scenes change (direction, speed, order, state) from video alone. To answer questions, they are joined to a language model trained on captions and question-answer pairs. But each caption or question names only a few attributes of a clip, so the model learns to retrieve those attributes rather than to read the clip, and the dynamics the encoder carries go unused: on the same items, a linear probe reads speed from frozen V-JEPA features at 87% while the VLM answers at chance. We show where dynamics lives in a frozen encoder: in a subspace shared across videos for predictive encoders such as V-JEPA, but not as a shared component of frame-wise, text-aligned encoders such as SigLIP. We then build counterfactual supervision that makes the language model read it: pixel transformations turn public videos into answer-balanced variants whose answers are set by the transformation and differ only in the change. Trained in a single stage on 4.53M public samples with 32 visual tokens per frame, where open video LLMs use several stages, 9M to 172M samples and 64 to 256 tokens per frame on average, the resulting V-JEPA-LM models (1B to 8B, Qwen3 and Llama) lead every size class in average score across eleven video benchmarks; our 4B model scores 72.1, against 61.1 for the best open 4B and 64.8 for the best open 8B model. A controlled encoder x supervision study shows that the encoder decides what can be read and the supervision what is read: at a matched budget, counterfactual supervision adds 4.5 points on average and 38 on the speed questions of natural benchmarks. Code and models: https://github.com/kkt94/Llama-V-JEPA-LM.

arXiv 原文

订阅 AI Pulse

每天 08:00 · 12:30 · 18:30 · 23:50 更新