AI 写代码能过测试,却完不成任务
现在的 AI 编程助手,单看测试通过率已经不错了——但把它放进一个真实的软件公司里,它立刻露馅。研究者造了一个模拟组织:26 个同事,各有分工、各有所知,AI 得从一句简短交接开始,自己去找需求、对信息、问对人、再动手改代码。结果最强的两个模型,能通过约 63% 的测试,却连 8% 的完整任务都交不出来。更扎心的是,把需求完整喂给它,测试通过率涨了 16 个百分点,任务完成率却纹丝不动。问题不在写代码,在于它不会追着需求跑:问过的问题不跟进,拿到需求也不做完,31% 自称完成的需求其实过不了测试。这不是你明天能用的工具,但它划出了一条分界线:AI 从『能写代码』到『能干活』,差的不是算法,是像人一样把一件事从头盯到尾的耐心。
📄 原文摘要(英文)
A coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains. Each assignment places an agent in a simulated engineering organization with 26 colleagues who have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. Starting from a brief handoff, agents must recover requirements distributed across colleagues and workplace records, reconcile conflicting evidence, and implement changes evaluated by private executable tests. We evaluate eleven models and find a substantial gap between partial implementation and complete task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments. Providing complete requirements upfront improves F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, yet leaves complete task success unchanged. Requirement-level trajectory analysis reveals failures throughout the delivery process: Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their accountable owners across runs but obtains only 64% of colleague-held requirements. Deferred questions go unfollowed, implementations remain incomplete even after requirements are acquired, and 31% of requirements reported as completed fail their tests. These findings highlight sustained requirement tracking, from clarification through implementation, verification, and handoff, as a key challenge for reliable autonomous software engineering.