最强AI造工具，一半时间在翻车
我们总在测AI能不能完成任务，这篇反过来测AI能不能给自己造工具。研究者让19个顶级模型读一份开发文档，在一个隔离环境里搭出完整的「工具壳」，再让这个壳去驱动AI跑328个测试。结果最强的GPT-6 Astra通过率53%，最弱的27%，差距26个百分点——但更扎眼的是，连最强的也有一半测试过不了，失败集中在「跨模块功能怎么拆分」这种工程组织问题上。它不是你明天能用上的东西，但它指出了一个趋势：AI的短板已经从「会不会答」转移到「会不会搭」，而搭工具这件事，连最强模型都还差得远。
📄 原文摘要(英文)
Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering.