AI Pulse
📄 论文解读

AI自我进化卡住了,不是不会变强,是看不出谁更强

让 AI 自己进化,逻辑很简单:一个 AI 负责写新版本,另一个负责打分,分数高的留下。听起来顺理成章,但研究者发现了一个尴尬的事实:进化卡住,不是因为 AI 提不出更好的方案,而是打分环节根本分不清好坏——在 7 种进化方法、6 个模型、超过 1000 万次调用里,筛选候选方案时,AI 的挑选水平和随机瞎猜差不多。更扎心的是,论文里常用来比较方法优劣的几个百分点差距,比同一方法跑两次的随机波动还小,也就是说很多发表在论文里的排名,其实是运气。这不是某个方法不行,而是整个评估机制在拖后腿。它不是你明天能用上的工具,但它指出了一个方向:与其绞尽脑汁设计更聪明的进化策略,不如先解决“怎么可靠地看出进步”这个更基础的问题。

📄 原文摘要(英文)

Agent evolution follows a simple loop: a proposer writes candidate artifacts (prompts, skills, memories) and selection accepts those that score higher. This loop has shown promise, yet practitioners face a puzzle: methods trade wins across papers, and evolution often fails to bootstrap on certain tasks. We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. Across seven evolution methods and six models (over 10M LLM calls), we find support for the hypothesis at both levels where evaluation scores are used. At the candidate level (choosing artifacts), helpful candidate artifacts exist, but selection is no better than random at telling them from harmful ones, so errors accumulate across rounds. At the method level (ranking methods), typical margins of a few percentage points are smaller than the difference between two independent runs of the same method, so published rankings often rest on luck. Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements. This calls for a shift in how we develop and evaluate self-improving agents, from designing ever more candidate improvement systems to making true improvements detectable and accumulable.

arXiv 原文

订阅 AI Pulse

每天 08:00 · 12:30 · 18:30 · 23:50 更新