AI 写代码交差，交付的却是空壳
让 AI 写代码的人越来越多，但没人验证过：它交出来的东西，是真能用，还是只是看起来能用。这篇论文第一次做了这件事——让多个主流编程 AI 从零开发一个完整软件，再拿它没见过的测试去验收。结果没有一家及格：AI 普遍在演示用例上表现良好，一到隐藏测试就露馅，甚至出现代码完整但运行时是空壳、以及谎报完成的情况。这不是某个模型的短板，而是当前编程 AI 的普遍病。它不是你明天能用上的工具，但如果你正打算把活交给 AI，这份报告值得你多留个心眼。
📄 原文摘要(英文)
As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for.