AI裁判只花千分之一钱，准确率只差3%
让AI当裁判评判另一个AI，一直是又贵又让人不放心的活。这篇发现，一个只输出结论、不给理由的轻量裁判，在普通偏好判断和事实核查上，准确率只比最强的大模型裁判低3个百分点，费用却只有对方的0.36%。差距主要出现在需要检查推导过程、或抵抗精心编造的误导答案时——而这些恰好是低置信度的判断。于是研究者设计了一个级联方案：轻量裁判有把握就直接定，没把握就升级给重裁判，最终保留了99%的准确率，成本还更低。它不是你明天就能部署的东西，但它指向一个方向：大规模评测AI时，不必每次都请最贵的裁判，先让便宜的筛一遍，把难题留给贵的。
📄 原文摘要(英文)
LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at scale. We study whether a decision-only judge can provide an economical first pass and identify when stronger evaluation is needed. Comparing jev-as-a-judge with sixteen generative and reward-model judges, with blinded human adjudication, we find it within three percentage points of a state-of-the-art LLM judge, our strongest comparator, on ordinary preference and evidence-grounded factuality at 0.36% of the comparator's fee. Larger gaps arise when judgments require checking a derivation or resisting an elaborately written wrong answer. On several benchmarks, JEV's gap to this comparator is concentrated in low-confidence decisions. A frozen cascade that accepts confident verdicts and escalates uncertain ones retains 99% of the comparator's accuracy at lower cost.