让AI自己造自己的工具，结果出乎意料
我们总在测AI能不能答对题，这篇反过来：测AI能不能给自己造一套答题系统。研究者让AI从零开始搭建自己的执行框架，再让它根据运行反馈自己改。结果很分裂：在写文章和机器学习实验上，AI自己造的工具能赶上甚至超过人类工程师精心设计的；但在写代码和搜索研究上，差距巨大。更麻烦的是，AI造的工具换一个模型来用，效果就大打折扣——它像是给自己量身定做的，别人用不顺手。这不是你明天能用上的东西，但它戳破了一个流行假设：AI越强就越会自己搭台子。至少现在，它只会给自己搭。
📄 原文摘要(英文)
As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.