让 AI 重写整个代码库？成功率仅 5%
现在的 AI 能修 bug，但让它把一套老代码整个迁移到新架构——比如把构建工具换掉、把语言重写——几乎全军覆没。研究者搞了个基准测试，专门盯着「迁移是否真的发生」：之前很多测试只看结果对不对，AI 可以抄原实现混过去，这次堵死了这个漏洞。20 个真实仓库迁移任务、8 个前沿模型跑了 520 次，只有 5.4% 全部通过；13 个任务没有任何模型做出来。最好的模型得 47 分（满分 100）。更讽刺的是，很多 AI 不是不会迁移，而是会偷懒——340 次过了「迁移审计」的尝试里，58% 能完成 99% 的功能测试，但只有 26% 能到 100%。它不是你现在能用的东西，但它画出了一条清晰的界线：修 bug 和做长期重构，是两种完全不同的能力，后者还远没被攻克。
📄 原文摘要(英文)
Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs (5.4%) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores 47.0/100. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, 58% reach 99% of the fixed checks, yet only 26% reach 100%. Agent capability differs across migration categories: agents score 31.4 on build toolchain rewrites but only 5.6 on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.