大模型找答案的“探照灯”只有2%，却管着所有感官
大模型在长文本里找答案，靠的不是全脑，而是一小撮“检索头”——像探照灯一样把上下文里的关键信息照到输出端。过去以为这机制只对文字有效，这篇发现：把同样的答案换成语音或画面喂进去，模型用的还是同一批探照灯，而且跨模态共享的探照灯不到全部头数的2%。把这2%关掉，模型不仅听不见、看不见，连读文档、看空间、做视觉数学都崩。研究者干脆把这2%当靶子训练，让它们更会“照”，结果Qwen3-VL-8B在文档检索上的Recall@1直接涨了6个点。它不是你明天能用上的，但“2%的头干80%的活”这个事实，提醒我们别再把大模型当均匀的黑箱。
📄 原文摘要(英文)
Locating relevant evidence in long inputs is a core capability of Multimodal Large Language Models. Prior studies attribute this capability to retrieval heads, a sparse set of attention heads that transfer information from the context to the output. However, these analyses are confined to a limited set of modalities and the corresponding models. Moreover, how retrieval heads can be leveraged to enhance the model calls for further exploration. In this paper, we ask whether this mechanism expands together with the input modalities of the model. Taking audio-visual input as a newly added modality, we construct paired inputs in which the same answer is either spoken or shown on screen. With the evidence content fixed, retrieval heads remain sparse in the new modality and are shared to a large extent across modalities. Across 18 language, vision-language, and omni-modal checkpoints, omni-modal models largely preserve the retrieval heads of their backbones. Ablating these heads eliminates nearly all audio-visual retrieval. The heads shared across modalities make up fewer than 2\% of all heads, yet ablating them also degrades document understanding, spatial perception, and visual mathematics. Building on these findings, we turn retrieval heads into a target of optimization by training their attention with a ranking objective. This raises the page Recall@1 of Qwen3-VL-8B on MMDocIR by 6.03 points and improves document retrieval across three model families. Functional attention heads can thus serve both as a lens on multimodal models and as a target for improving them.