AI 能当运筹顾问吗?最贵考题:310 万美元的真实项目
给 AI 的考题,终于从「标准答案」换成了「真实账单」。研究者把 10 个行业、总价 368 万美元的付费咨询项目原样丢给 11 个前沿大模型,要求它们直接读客户的原始材料、自己问问题、自己出方案——而不是像以往那样拿到整理好的数学题。结果:论方案质量,几个模型已经能超过人类专家;但论「交得出能落地的东西」,最强模型也只有 87% 的专家水平,因为 AI 经常在可行性上翻车——方案漂亮,客户却用不了。更值得玩味的是,AI 在「追问客户」这件事上表现不错,但问得最勤的阶段不是一开始,而是方案做到一半、发现问题时。它不是你明天能用的:这更像一张「AI 顾问能力体检表」,告诉你大模型离真正接活还差在哪儿——不是不会算,而是不会问、不会保证交付。
📄 原文摘要(英文)
Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.