给AI当裁判的模型，反而让AI更贵了
给AI当裁判的模型，反而让AI更贵了。研究者造了一个专门的「决策模型」Jev，只在几个固定节点替主模型做小决定：要不要停、要不要检查、要不要换思路。听起来合理——把小事交给便宜模型，大事留给贵模型。结果在21个任务、315次试验里，成本涨了19%，成功率纹丝不动(都是73%)。更讽刺的是，离线测试里Jev判断「该不该收工」的排序能力其实更强，但校准一塌糊涂：它说有50%概率完成的任务，实际87%都通过了——也就是说它老让主模型去复查已经做对的事。没有哪个控制器选择过放弃任务。一个能分清好坏信号的模型，变成干预手段后反而帮倒忙。这不是你明天能用上的东西，但它戳破了一个流行假设：给AI加个「监督者」不一定省钱，判断力好不等于干预得好。
📄 原文摘要(英文)
Agent scaffolds increasingly delegate small decisions (whether to stop, verify, or change approach) to cheaper models. We ask whether a typed decision model, Jev 1.13, consulted only at deterministic decision points, makes a Claude Sonnet 5.5 terminal agent cheaper without making it worse. In a pre-specified study on Terminal-Bench 2.1, we hold one agent loop fixed and vary only the controller: none, deterministic rules, a small LLM (Claude Haiku 5.5) answering the same questions, Jev, and a placebo that samples Jev's development-phase choice frequencies at the same moments. On 21 held-out tasks (315 trials), Jev raised cost per attempt by 19% (geometric-mean ratio 1.19, 95% CI 1.02-1.36) at an identical pass rate (73.0% vs. 73.0%). Jev was cheaper than the small-LLM controller (ratio 0.81) but 22% more expensive than the placebo, and no pass-rate difference between arms was detectable. Offline, Jev ranked finish requests better than the alternatives (AUROC 0.78 vs. 0.62 for rules and 0.60 for the small LLM) but was poorly calibrated: its mean probability that a run was complete was 0.50 when 87% of those runs passed, so it asked the agent to verify work that was already correct. No controller ever chose to abandon a run. Informative decision signals did not become useful interventions; we discuss what that would require.