科学代码修不好，不是AI笨，是它不懂科学
AI写代码已经很能打了，但让它修科学软件——比如分析天文数据、模拟分子动力学的代码——它立刻露怯。新基准SWE-bench Science把119个真实bug丢给当前最强模型，结果最好那个也只修好不到一半。研究者拆了四种死法：不懂科学概念、只改表面不修根、修了这头漏了那头、不会举一反三。更扎心的是，给AI喂科学知识不一定帮倒忙：靠谱的指引能帮它收窄搜索范围、省token;但给错了方向，它反而死磕一条路，修得更差。这不是你明天能用上的工具，但它是你该知道的信号：科学软件修bug这件事，AI离靠谱还远，别把实验结论押在它身上。
📄 原文摘要(英文)
Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce SWE-bench Science, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, Claude Code with Opus-5 (max), achieves a pass@1 below 50\%, highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.