AI Pulse
📄 论文解读

AI看图时,脑子里也有个「工作台」

我们习惯把大模型当黑箱,这篇却给它做了一次「意识探测」:借认知科学里的全局工作台理论,看视觉语言模型内部是不是也在做同一件事——从一堆并行信息里挑出当前任务要用的概念,放进一个共享空间,再交给下游去推理。他们造了个叫 MC-Lens 的工具,把模型的视觉编码器和语言解码器中间状态都映射到同一个「词表语义空间」,于是能直接读出模型此刻「想到」了哪些概念,还能动手改这些概念、看答案会不会跟着变。结果有三层:概念确实可读,而且会随任务选择性浮现;被读到的概念真的参与后续推理,不是摆设;这种「按需取用」的能力,比「概念能被读到」出现得更晚——先有货,后才会挑。跨不同模型家族和规模,这套模式基本一致。它不是你明天能用上的东西,但给「AI 内部有没有类似注意和工作记忆的结构」提供了一个可实验的抓手。

📄 原文摘要(英文)

Vision-language models (VLMs) integrate visual and textual information, but their outputs do not reveal internal concept representations. Global workspace theory (GWT) proposes that information is selected from parallel brain processes, integrated in a shared workspace, and made available to specialised processes. We introduce the Multimodal Concept Lens (MC-Lens), a multimodal Jacobian lens, to study these functions in VLM concept processing. Its vision encoder and language decoder lenses map intermediate states to a shared semantic space defined by the decoder's output vocabulary to compare concept accessibility across modalities. MC-Lens supports internal concept interventions to examine the role of accessed concepts in downstream computation. (i) Accessibility and selective access. Multimodal concept information is readable in intermediate states and selectively accessible as tasks require. We extend "readout carriers" from one name token to a token set to better capture and express concept information and support counterfactual experiments. (ii) Concepts in downstream computation. Counterfactual experiments show that accessed concept information participates in downstream computation and reasoning. (iii) Development of cross-modal alignment and selective access. We develop two J-space metrics tracking the development of cross-modal concept alignment more clearly than hidden state cosine in controlled VLM training. The experiment also shows that reliable selective access develops later than concept accessibility. (iv) Shared functions across models. Across VLM families and sizes, we find shared patterns of concept accessibility, query-dependent selectivity and cross-task intervention effects despite differences in internal organisation. These findings offer an experimental basis to study GWT in multimodal artificial systems and how it can guide cross-modal alignment improvements.

arXiv 原文

订阅 AI Pulse

每天 08:00 · 12:30 · 18:30 · 23:50 更新