AI 科学家是假的：它靠技巧刷分，不是真懂科学
前沿 AI 被吹成「自动科学家」，这篇给了它一场野外考试：在 20 个科学 ML 基准上，让 AI 去改进人类已发表的最好模型。结果很微妙——它确实在每个基准上都赢了，平均提升 10.4%，也没作弊。但拆开看，这些提升几乎全来自调参、换优化器这类工程套路，没有一条是研究层面的洞察。换句话说，AI 在刷分，不是在搞懂科学。研究者还发现三个老毛病：对「什么算新」没概念、不会用足算力、很快陷进局部调参的循环。有意思的是，给 AI 配上「研究构思」「实验管理」「检查清单」这类轻量插件，提升直接翻倍(Claude 从 10.4% 到 20.1%)，而且干预更多样、更贴近领域。这说明瓶颈不在模型聪明不聪明，而在它缺一套「做研究」的流程。它不是你明天能用上的工具，但它是衡量「AI 到底会不会做科研」的第一把尺子。
📄 原文摘要(英文)
Frontier coding agents are increasingly framed as end-to-end automated researchers, and ML for science offers a demanding test of that promise: improving a scientific model often requires domain insight beyond routine ML engineering. We introduce Agent4Sci, a long-horizon testbed for measuring how far today's agents can improve the published best models on scientific ML benchmarks, spanning 20 benchmarks across 11 domains. Rather than ranking agents on a leaderboard, Agent4Sci asks what kinds of progress agents can make on these tasks under in-the-wild conditions, and where they fall short. Empirically, today's frontier agents advance the published best on every benchmark, with Claude Opus reaching 10.4% mean relative improvement and no run flagged for academic misconduct. These gains, however, come almost entirely from a bag of ML-engineering tricks rather than research-level insight. Trajectory analysis surfaces three recurring limitations in this setting: novelty miscalibration, resource under-utilization, and rapid collapse into local engineering sweeps. Lightweight plug-in skills for research ideation, experiment management, and scheduled checklists raise Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, while yielding more diverse, domain-relevant interventions. Agent4Sci gives the community a way to track and push frontier agents from ML engineering toward domain insight on real-world ML-for-science problems. Code and data are available on GitHub (https://github.com/maszhongming/agent4sci) and Hugging Face (https://huggingface.co/datasets/MingZhong/agent4sci).