AI 能解难题，却搞不定真实客户
现在的 AI 评测，题目都是写好的：需求明确、边界清楚，考的是解题能力。但真实世界的交付不是这样——客户自己都说不清要什么，系统是活的，还有一堆没人告诉你的隐性约束。这篇论文造了个新考场 FDE-Gym:95 个来自真实项目的交付任务，AI 要跟模拟客户来回沟通、在像生产环境的系统里动手改，最后按「客户的业务指标提升了多少」打分。结果很扎心：最强的模型平均只拿到 0.56 分，95 个任务里只有 14 个真正搞定。最大的瓶颈不是能力，是「发现没说出口的需求」——一旦没有明示指引，成绩断崖式下跌。更麻烦的是，就算把关键信息都问到了，23 个任务里也只有 10 个能真正落地。这提醒我们：AI 在标准考题上的高分，离「替客户把事办成」还有一条鸿沟。它不是你明天能用上的东西，但解释了为什么很多 AI 项目在 demo 里惊艳、到客户现场就翻车。
📄 原文摘要(英文)
Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.