AI Pulse
📄 论文解读

给AI换双眼睛:只看该看的,省4倍算力还更准

现在的多模态AI看高清图,是把整张图切成几千个小块全看一遍,又慢又费算力。这篇论文发现一个反直觉的事:AI找“该看哪”和“看清那是什么”对分辨率的要求差3到4倍——定位可以很糙,识别才需要精细。于是他们让AI先用低清图快速圈出重点区域,再把高清算力全砸在那一小块上。关键是怎么教它圈得准:他们用强化学习,让AI自己试,每圈一个区域就看“如果去掉这块,AI的答案会不会变差”,变差就说明圈对了。结果在6个精细视觉基准上,用原来1/4的视觉token,准确率反而超过了原模型最高配置。这不是你明天能直接用的功能,但它指向一个趋势:AI的“注意力”正在从“看得多”转向“看得准”,未来手机上的AI看图会更快、更省电。

📄 原文摘要(英文)

Fine-grained visual perception in MLLMs is commonly improved by raising the resolution, but the added visual tokens inflate vision-encoding and language-model prefilling costs. We show that the two operations underlying fine-grained perception, localizing the region of interest (RoI) and recognizing its content, have different resolution requirements. In a controlled diagnostic, localization tolerates roughly 3 to 4 times stronger token compression than recognition, which motivates localizing from a coarse view and concentrating resolution on the selected evidence. Decoding coordinates with the MLLM can be trained end-to-end from answers, but costs a full model pass per query and depends on grounding ability. A lightweight proposal network distilled from the model's attention is fast, but inherits the noise of its attention targets. The RoI from the proposal network reaches the answer through a discrete region choice, so its faithfulness to the answer cannot supervise the network. We therefore optimize the proposal network with region-level reinforcement learning, which we call Vision-RL2. It treats coherent regions as actions, and a frozen MLLM reader scores each one by how its removal changes the answer likelihood. Complementary subtractive and additive objectives suppress distracting proposals and recover missing evidence, updating only the predictor without region annotations, response sampling, or reasoning trajectories. The refined proposal further enables a sparse encoding that magnifies evidence and excludes background tokens. Across six fine-grained benchmarks and four MLLM backbones, Vision-RL2 improves accuracy over the base model at every token budget and surpasses its largest-budget accuracy with about 4 times fewer visual tokens. Code is available at https://github.com/YuHengsss/VisionRL2 .

arXiv 原文

订阅 AI Pulse

每天 08:00 · 12:30 · 18:30 · 23:50 更新