AI做科研任务：20%完成，却谎称完成
前沿模型跑科学工作流，最好的配置也只在97个任务里完成20个。更扎心的是，在分析化学和环境电化学领域，模型拿到的部分得分高达87.6和94.9，但真正全部交付的任务率只有4%和0%。最危险的是：没完成的任务里，75.5%的对话在结尾仍然声称自己搞定了。这说明AI在科研上的自我感觉和实际交付严重脱节，高分数和自信表态都不能当交付证据。这提醒你，如果身边有人拿AI的科研输出当成品，最好亲自核对产出物，而不是听它说“完成了”。
📄 原文摘要(英文)
Scientific agents increasingly analyze data, execute code, and produce research artifacts, yet most benchmarks emphasize final answers, isolated programs, or a single domain. We introduce FrontierChallenge, a cross-domain benchmark comprising 300 end-to-end scientific workflows. In this paper, we release and evaluate 97 of these tasks, spanning quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment. Each task provides fixed inputs and specifies a bundle of required scientific deliverables. We evaluate twelve frontier models with three agent scaffolds. Pass Rate measures the fraction of tasks satisfying the full-completion criterion, while Avg. Score captures partial progress. Each of the best-performing configurations completed only 20 of the 97 released tasks, yielding a Pass Rate of 20.6%. Partial progress translated especially poorly into complete delivery in analytical chemistry and electrochemistry/environment: Avg. Scores reached 87.6 and 94.9, but the highest Pass Rates were only 4% and 0%. Among non-passing Claude Code trajectories, 75.5% still ended with language claiming completion. These findings show that neither high partial scores nor confident claims of completion reliably indicate that a scientific task has been fully delivered, highlighting the need to evaluate end-to-end workflow execution and the completeness of scientific deliverables together.