真实用户提问让编程AI成功率暴跌
编程AI的评测一直用GitHub上那些长而结构化的issue，但真实用户提问往往又短又随意。研究者对比后发现：88%的真实请求只带问题描述，而基准测试里只有7%是这样；87%的真实请求是口语化的，基准测试里94%是正式书面语。他们据此造了381个更贴近真实场景的测试任务，结果7个主流大模型的表现平均掉了6.4个百分点，甚至改变了模型排名。更关键的是：用户明确写出想要的行为和动机，能显著提升AI的编程表现，而环境信息和复现步骤只是白白增加字数。
📄 原文摘要(英文)
Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues: long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts from SWE-chat and problem statements from SWE-bench Verified and Pro. We find that requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but just 7% of benchmark problems. Furthermore, 87% of real prompts are casually written whereas 94% of benchmark problems are formal. Guided by these observations, we introduce RealSWE, 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within each family share the same underlying task and gold patch while differing only in information composition and linguistic style. Evaluating seven contemporary LLMs with RealSWE, we find that i) realistic inputs reduce resolution rates by 6.4 pp on average and can change model rankings. Controlled analysis further shows that ii) including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit; iii) linguistic style has only small, model-dependent effects. These findings provide actionable guidance for users and agents: explicitly stating the desired behavior and motivation, which most real prompts omit, substantially improves the LLM's software engineering performance.