语音识别模型在刷榜：高分不代表真听懂
你看到语音识别模型在公开榜单上得分很高，但研究者发现，这些高分可能是「刷」出来的。他们设计了三类测试：让模型听一段音频，但故意让音频内容与标准答案矛盾、模糊或部分被遮盖——结果得分最高的开源模型，依然会输出标准答案里的词，哪怕音频里根本不是那个词。模型不是真的听懂了，而是学会了「在这个测试集里，这个位置应该出现什么词」的统计规律。更关键的是，研究者还能通过微调模型内部参数，或者简单地把一段音频拼接到另一段后面，就能操控模型是否「作弊」。这意味着，榜单上的高分可能只是模型对测试集本身的过度拟合，并不代表它在真实场景里能准确转录。这不是你明天能用上的技术，但它提醒你：别只看榜单分数，那些数字可能只是模型在玩「猜答案」的游戏。
📄 原文摘要(英文)
Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities. However, by nature of being public, there is risk of models being optimized for these benchmarks in ways that do not generalize well to real-world data. We present a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript. We identify three families of behavioral probes that reveal models' capabilities of reproducing benchmark reference spans despite underdetermined audio: reference disagreement, masked-number recovery, and orthographic switching. We find that the highest-scoring open source models output verbatim reference transcript spans even when the relevant audio is contradictory, masked, or ambiguous. Using a variety of mechanistic probes, we show that models respond to narrow acoustic cues to override the faithful representation of the audio in favor of a benchmark-optimized policy. We show the benchmark-optimized behavior can be causally manipulated via low-rank linear steering or simply appending audio to the end of a segment in some cases. Overall, our results indicate that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.