AI写代码的“高分”可能是背题
现在衡量AI写代码能力的基准测试，用的都是GitHub上最热门的开源仓库——而这些仓库大概率早就被AI“背”过了。研究者做了个实验：把测试仓库的变量名、文件结构、代码写法全部打乱重写，但保证功能完全不变。结果所有主流AI模型的成绩都明显下滑，而且要多花不少步骤才能找到该改的地方。这说明它们的高分里，有一部分靠的是“见过这道题”，而不是真的会读代码、找问题。它不是你明天能用上的东西，但它提醒你：AI写代码的排行榜，水分可能比想象中大。
📄 原文摘要(英文)
Repository-level coding benchmarks have become the standard for evaluating coding agents, yet they inherently suffer from data leakage because they are built upon popular open-source repositories repeatedly used for training. Consequently, strong performance may reflect memorization of canonical repository cues rather than robust repository reasoning. We propose SchrodingerRepo (Schrödinger's Repository), an evaluation framework for testing coding agents under dynamically instantiated repository representations. Instead of repeatedly using a static representation of the test repository, SchrodingerRepo treats the test repository as an evaluation-time latent variable that is dynamically instantiated only when the agent enters the evaluation environment. The instantiated repository preserves the original executable behavior while eroding familiar cues such as naming conventions, file layouts, and implementation patterns through four transformation levels: problem statement reconstruction, namespace remapping, intra-file layout reordering, and functionality-preserving code rewriting. We evaluate popular LLMs on SWE-bench Verified and SWE-QA. Results show that removing familiar repository cues consistently degrades agent performance and substantially increases interaction costs across models. Further analysis reveals that the additional cost is primarily caused by increased difficulty in repository exploration and localization. These findings suggest that current coding agents may partially rely on memorized repository-side cues, highlighting the need for evaluation under dynamically instantiated repository representations.