AI评测的皇帝新衣：高分模型一考细节就崩
现在的AI评测有个怪现象：模型在标准测试里拿高分，但一放到真实场景就露馅。这篇论文直接戳破这个泡沫——他们设计了一套“扣分制”考试：先让AI描述一张信息密集的图片，然后逐条核对它是否答对关键事实（比如“图里有几只猫”），以及是否漏掉细节点（比如“猫的项圈是红色的”）。结果发现，很多模型能说对零散元素，但一旦要求“所有事实必须同时正确”，就集体翻车。更扎心的是，开源模型和闭源模型之间始终存在8%的感知差距，这和当前“开源追上闭源”的流行说法相反。这套方法不是给你明天用的工具，但它提醒你：别被高分榜单骗了，AI的“看见”和“看清”之间还有鸿沟。
📄 原文摘要(英文)
We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived from golden captions constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 8% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.