真实用户提问让编程AI成功率暴跌
编程AI在标准测试里看着很强，但真实用户根本不会像测试题那样写需求。研究者对比了真实提问和基准测试题：88%的真实请求只有一句问题描述，而基准题里只有7%这么写；87%的真实请求是随口写的口语，94%的基准题是正式书面语。他们据此造了381组「同一任务、不同问法」的测试，发现真实问法让AI成功率平均掉6.4个百分点，甚至改变模型排名。最有用的发现：用户明确写出「想要什么行为」和「为什么」,AI表现显著变好；而写环境信息和复现步骤只是多费token，没实际帮助。对普通人的直接含义：以后让AI改代码，别只丢一句「这个不对」，把期望结果和动机说清楚，它就能干得更好。
📄 原文摘要(英文)
Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues: long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts from SWE-chat and problem statements from SWE-bench Verified and Pro. We find that requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but just 7% of benchmark problems. Furthermore, 87% of real prompts are casually written whereas 94% of benchmark problems are formal. Guided by these observations, we introduce RealSWE, 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within each family share the same underlying task and gold patch while differing only in information composition and linguistic style. Evaluating seven contemporary LLMs with RealSWE, we find that i) realistic inputs reduce resolution rates by 6.4 pp on average and can change model rankings. Controlled analysis further shows that ii) including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit; iii) linguistic style has only small, model-dependent effects. These findings provide actionable guidance for users and agents: explicitly stating the desired behavior and motivation, which most real prompts omit, substantially improves the LLM's software engineering performance.