AI裁判新思路：没把握就上交，省钱还准
让AI当裁判评判别的AI，一直是又贵又让人不放心的事。这篇提出一个反直觉的做法：只让便宜的AI做“决定”，不解释理由，没把握就直接上交。结果，这个“哑巴裁判”在最常见的偏好判断和事实核查上，准确率只比最强AI裁判低不到3个百分点，成本却只有对方的0.36%。差距主要出现在需要检查推导过程或识破精心编造的错答案时，而这些恰恰是它自己承认没把握的地方。于是研究者设计了一个“冻结级联”：有把握就拍板，没把握就升级，最终保留了最强裁判99%的准确率，成本还更低。它不是你明天就能用的工具，但给“怎么让AI评估AI”指了一条省钱又诚实的路。
📄 原文摘要(英文)
LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at scale. We study whether a decision-only judge can provide an economical first pass and identify when stronger evaluation is needed. Comparing jev-as-a-judge with sixteen generative and reward-model judges, with blinded human adjudication, we find it within three percentage points of a state-of-the-art LLM judge, our strongest comparator, on ordinary preference and evidence-grounded factuality at 0.36% of the comparator's fee. Larger gaps arise when judgments require checking a derivation or resisting an elaborately written wrong answer. On several benchmarks, JEV's gap to this comparator is concentrated in low-confidence decisions. A frozen cascade that accepts confident verdicts and escalates uncertain ones retains 99% of the comparator's accuracy at lower cost.