AI 解释器也会选择性失明
我们以为 AI 解释器是中立读心器，结果它也会被训练带偏。研究者让一个模型心里藏着某个词、嘴上不说，再训练另一个模型去读它的内部状态。按理说，读心器该成为这个秘密的专家；实际它反而对训练时反复出现的那个词选择性失明——词明明还在它脑子里，解码器也能解出来，就是读不出来。这说明 AI 解释器不是透明玻璃，而是有自己盲区的学习系统。它不是你明天能用上的，但提醒你：AI 解释 AI，也可能看走眼。
📄 原文摘要(英文)
Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.