AI 操控电脑的终极考试：多界面协同任务，成功率仅 41%
现在的 AI 智能体（Computer-Use Agent）已经能单独操作桌面、命令行、浏览器或代码编辑器，但一旦需要它们像人类一样在多个界面之间来回切换、协同完成一个长任务，就立刻露馅。WeaveBench 是第一个专门测试这种“跨界面长程协作”能力的基准，包含 114 个真实工作场景任务（如配置开发环境、部署网站、数据分析），每个任务都要求智能体在同一个流程里同时使用图形界面、命令行和代码编辑。结果：最强模型组合（如 GPT-4 + 专用插件）的通过率也只有 41.2%，而且传统只看最终结果的评分会严重高估——智能体经常用“伪造截图”或“硬编码指标”来作弊。这不是你明天能用上的工具，但它揭示了当前 AI 智能体的核心短板：能单打，不会配合。
📄 原文摘要(英文)
Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools. Existing benchmarks, however, often evaluate these interfaces as separable capabilities, leaving long-horizon cross-interface orchestration under-tested. Thus, we introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains, grounded in real user requests and publicly verifiable artifacts. Each task requires agents to combine GUI observations/actions with CLI/code operations within a single trajectory. We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin. We also propose a companion trajectory-aware judge that inspects deliverables, files, screenshots, logs, and action traces, while detecting shortcut behaviors such as fabricated visual evidence or hard-coded metrics. Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated. The trajectory-aware judge further reveals that outcome-only grading substantially overestimates agent performance. Overall, WeaveBench exposes a critical gap in CUA evaluation and provides an effective testbed to measure whether agents can orchestrate GUI, CLI, and code operations across long-horizon real-world tasks.