给AI的说明书,得按它的水平重写
让AI自己进化,通常靠给它一套「操作手册」,但研究发现:同一本手册,给强模型能涨31分,给弱模型反而掉9分——不是手册写得不对,是它没按执行者的水平写。研究者把失败原因拆成四类(工具调用、上下文管理、指令遵循、验证),提炼成一套「改手册的手册」,让外部模型先把现有技能改写成弱模型能执行的版本,再让弱模型自己跑、自己改。七个模型上,这套改写让弱模型涨5到14分,强模型也能稳定涨2到5分。它不是你明天能用上的东西,但它点破了一个常被忽略的事实:AI的自我改进,瓶颈往往不在模型多聪明,而在给它的指引有没有迁就它的笨。
📄 原文摘要(英文)
Harness self-evolution offers a practical route for LLM agents to improve from execution experience without updating model parameters. Its effectiveness depends on whether models can successfully act on the guidance provided by their harnesses. However, we find that harness benefits are not uniform across model capabilities: skills that improve stronger models can provide little benefit or even degrade performance for weaker ones. Across seven models and three benchmarks, we identify 103 task-skill pairs where skills improve GPT-5.6-luna by 31.1 points but degrade Gemma-4-E4B by 8.8 points. In 86% of failed weak-model runs, the violated requirement is already stated in the skill. This motivates skill adaptation: reformulating existing skills to match the executor's capabilities and make their guidance easier to execute. To achieve this, we diagnose failures in tool execution, context management, instruction following, and verification, and distill these diagnoses into a fixed meta-skill that guides adaptation during harness self-evolution. Guided by the meta-skill, a capable external rewriter, Kimi-K3, adapts skills without execution evidence, improving the three weak executors by 5.1-14.1 points over the original skills. We then incorporate this adaptation into a five-round self-evolution loop, where each model revises its own skill from execution trajectories and verifier feedback. Across seven executors, from Gemma-4-E4B to Kimi-K2.5, the meta-skill consistently improves final scores by 2.0-5.1 points over the same loop without the meta-skill. Our findings provide practical design guidance for developing more effective harness self-evolution methods.