AI 智能体评测：别再拿沙盒糊弄了
现在的 AI 智能体评测大多在沙盒里做单轮问答，跟真实世界差太远。这篇搞了个新基准 UniClawBench，把智能体拆成 5 种基础能力（技能使用、探索、长上下文推理、多模态理解、跨平台协调），然后在真实的 Docker 容器里跑任务，用多轮闭环评估——有执行者、监督者和用户三个角色模拟真人反馈。结果发现，最强的模型在真实任务上成功率也不到 40%，而且框架设计（比如怎么调用工具）对表现的影响有时比模型本身还大。它不是你明天能用上的，但给行业指了个方向：别再只看模型参数，得看它在真实环境里到底能不能干活。
📄 原文摘要(英文)
The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments. To facilitate future research, we make our benchmark and code publicly available at https://github.com/HKU-MMLab/UniClawBench.