AI写代码的“高分”可能是背答案
现在的AI编程助手在基准测试里拿高分，可能不是真会写代码，而是“背”过这些开源仓库。研究者把测试仓库做了四层“易容”:改写题目、重命名所有变量和函数、打乱文件内部顺序、重写代码但保持功能不变。结果所有主流模型成绩明显下滑，交互次数大幅增加——它们确实在靠记忆仓库的“长相”来答题，而不是靠理解逻辑。这提醒我们：别把榜单分数当成真实能力，它更像“见过多少题”。
📄 原文摘要(英文)
Repository-level coding benchmarks have become the standard for evaluating coding agents, yet they inherently suffer from data leakage because they are built upon popular open-source repositories repeatedly used for training. Consequently, strong performance may reflect memorization of canonical repository cues rather than robust repository reasoning. We propose SchrodingerRepo (Schrödinger's Repository), an evaluation framework for testing coding agents under dynamically instantiated repository representations. Instead of repeatedly using a static representation of the test repository, SchrodingerRepo treats the test repository as an evaluation-time latent variable that is dynamically instantiated only when the agent enters the evaluation environment. The instantiated repository preserves the original executable behavior while eroding familiar cues such as naming conventions, file layouts, and implementation patterns through four transformation levels: problem statement reconstruction, namespace remapping, intra-file layout reordering, and functionality-preserving code rewriting. We evaluate popular LLMs on SWE-bench Verified and SWE-QA. Results show that removing familiar repository cues consistently degrades agent performance and substantially increases interaction costs across models. Further analysis reveals that the additional cost is primarily caused by increased difficulty in repository exploration and localization. These findings suggest that current coding agents may partially rely on memorized repository-side cues, highlighting the need for evaluation under dynamically instantiated repository representations.