AI偶尔成功一次，不等于可靠
现在的AI评测都在看它能不能答对、能不能调对工具；这篇说，真正干活不是这样。它搭了个沙盒，让AI在零售、酒店、保险、银行IT这些真实业务流里跑完整流程：要自己问缺的信息、守公司政策、协调好几个工具、最后把系统状态改对还不能留副作用。507个流程测下来，最强模型一次通过率65%，但试20次也只有25%能稳定成功——而且很多失败案例里，AI的每一步动作看起来都正常，只是最终状态错了。也就是说，你盯着它干活觉得没问题，结果账是错的。它不是你明天能用上的东西，但它解释了为什么AI做业务自动化总差一口气：偶尔成功和可靠，是两回事。
📄 原文摘要(英文)
Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, Thinkingbox-bench contains 507 policy-conditioned workflows across numerous scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Across proprietary and open-weight models, the strongest achieves 65.36% pass@1, but only 25.25% pass^20. Moreover, many failed trials show clean termination and valid state-changing actions, showing that response or tool-call-level signals are not clear proxies for end-to-end task completion. Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both Thinkingbox and Thinkingbox-Bench: https://github.com/microsoft/thinkingbox