AI 能自己跑实验，但判断力还不及格
大模型代理已经能连续几个月自主跑科研实验，几乎不用人盯。但真正的自主科研还需要「判断力」——什么时候该改方向、哪个结果值得追。研究者用 6 个月里 26 个真实项目、248 次人类专家出手纠正 AI 的记录，做成了一套判断力考卷：给 AI 看纠正前的项目记录，让它选「原计划」还是「专家改过的方案」。结果：五个顶级模型里四个正确率低于瞎猜，最高分 63.5%，而新入职的研究人员能到 73.4%。更反直觉的是，给模型看项目记录反而让它更坚持原计划、错得更稳。这不是你明天能用上的东西，但它划出了一条线：AI 做实验的执行力已经够强，真正卡住自主科研的，是它还不具备「知道自己在做什么、为什么这么做」的科学判断。
📄 原文摘要(英文)
Large language model (LLM) agents can keep scientific experiments running for months with little human supervision, but reliable autonomous research also requires scientific judgment. Evaluating scientific judgment is difficult because research decisions are open-ended and depend on project context. We construct SciΠvot, a 248-case benchmark of scientific judgment based on human corrections. The benchmark draws on records from six months of research in which senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly intervened to correct the agents' research judgments. In each case, an agent chooses between the original LLM plan and the human-corrected alternative, using only the project record available before the correction as context. We compare LLM and human performance and examine the effects of reasoning effort, access to the project record, and reconsideration. At medium reasoning effort with the project record, models systematically prefer the original LLM plan over the human-corrected alternative. Four of five frontier models perform below chance on the selected cases. The highest accuracy in the project condition across the tested models and reasoning effort settings is 63.5%, compared with 73.4% for fresh researchers. At medium effort, every model achieves lower accuracy with the project record than without it, and reconsideration does not consistently improve overall accuracy. Even with a human-corrected alternative available, frontier models struggle with the selected research decisions, while SciΠvot supports systematic evaluation of scientific judgment for LLM agents.