AI 科学家？最强模型也只在 17.8% 的任务上超越人类
我们总说 AI 能搞科研，但 Nature 论文的顶级成果，AI 真的能复现吗？这篇论文用 90 个来自 Nature 系列期刊的真实科学任务测试了当前最强的 AI 编程智能体，结果发现：即使是最强模型，也只在 17.8% 的任务上超越了论文中的原始结果。更关键的是，AI 的成功主要靠把科学问题翻译成熟悉的预测任务，而不是真正的科学创新。失败的原因不是理解不了任务，而是选错了方法或算力不够。它不是你明天能用上的工具，但它诚实告诉你：AI 离独立做科研还差得远。
📄 原文摘要(英文)
We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench