AI编程评测跑偏了：真实程序员根本不按套路出牌
AI编程助手好不好用，现在都用GitHub issue当考题：给一个bug描述，看它能不能修好。但研究者扒了4782段真实程序员在IDE里的操作记录，发现真实使用根本不是这样——33%的会话里，用户至少发三条消息，而且这些长会话里用户会来回切换任务：问代码、做规划、审查、重构、跑程序，混着来。拿issue当考题，等于只考了真实工作流里的一小块。更麻烦的是，不同来源的交互数据，任务切换模式差异巨大，根本不存在一个“标准”的评测流程。研究者提出一个方法，能把现有基准改造成匹配特定使用场景的交互模式，但初步实验发现：把任务拆成多步让AI逐步解决，成本翻倍，成功率却没稳定提升。这说明评测协议本身就会影响结果——你测出来的“强”，可能只是考题出得巧。
📄 原文摘要(英文)
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories. In a pilot on 700 SWE-Bench Pro tasks, solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate: the interaction protocol itself is an important dimension of evaluation.