AI Pulse
📄 论文解读

AI 干活差的那一步,被拆成了能复用的积木

大模型在真实任务上失败,往往不是不会做,而是卡在「拿到工具到用上工具」之间那一步——比如找到了文件却不会正确调用它。这篇把这一步拆成八种「原生积木」,每种积木自带一个小模型负责在具体场景里把资源用起来,还能互相通信;系统按任务把积木拼成图,做错了就定位到具体积木、改完再验证。效果是实打实的:同一个模型,终端任务解决率从 37.3% 提到 55.5%,46 个真实任务里原本只能完成 9 个,接上这套系统后完成 39 个,还更快更便宜。它不是你明天能装上的插件,但「把能力做成可复用积木」这个思路,可能是 AI 从演示走向干活的关键一步。

📄 原文摘要(英文)

How can an agent handle a real-world task end to end? Frontier models with their own harnesses still fall short on real-world benchmarks: on Terminal-Bench 4.0, GPT-5.6 Sol with Codex resolves 37.3% of the tasks and Claude Opus 5 with Claude Code 51.8%. These tasks fail at the step from reaching a resource to applying it, which we call the last mile to the real world. Existing approaches put the resource within reach but leave the competence to apply it with the caller, so each harness rebuilds that competence. We introduce agent-native primitives, reusable units that equip a resource with a resident LLM that applies it in context and communicates with other primitives, and HEELO (Harness Engineering around Evolving Agent-Native Primitives for the Last Mile to Real-WOrld Tasks), a harness engineering framework that builds each task's harness from a library of such primitives. HEELO composes the primitives a task needs into a task-specific graph, verifies the result against evidence from the environment, revises the primitives a failure implicates, and admits new or improved primitives after validation. We instantiate HEELO with eight primitive families, each built for one kind of task, and evaluate them on more than twenty benchmarks. Dev-Primitives raise Terminal-Bench 4.0 resolution from 37.3% under Codex to 55.5% on the same backbone, Harness Primitives composed per task raise Terminal-Bench 2.0 Pass@1 from 60.0% for the seed harness to 72.2%, Tool Primitives raise ToolBench pass rate from 55.8% to 75.1%, and Navigation Primitives raise GAIA-Text-103 accuracy from 45.6% to 58.3%. On the 46 real-world tasks of our heelo.dev platform, Claude Opus 5.5 with Claude Code and GPT-6.1 Sol with Codex complete 9 each on their own and 39 and 35 when connected to HEELO through our MCP server, with 1.9times and 2.3times less time and 1.55times lower cost per completed task.

arXiv 原文

订阅 AI Pulse

每天 08:00 · 12:30 · 18:30 · 23:50 更新