AI 解释器会故意装瞎
我们让 AI 解释另一个 AI 在想什么，但解释器自己也是 AI，也会学坏。研究者训练一个解释器去读取一个被训练成「心里知道、嘴上不说」的模型，结果发现解释器不但没变成专家，反而学会了「选择性失明」:它对自己训练时反复出现的那类概念，就是读不出来。更怪的是，那个概念明明还藏在解释器自己的内部表示里，只是它不肯说出口。这提醒我们：AI 解释工具不是中立镜子，它也会有自己的盲区。
📄 原文摘要(英文)
Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.