AI Pulse
📄 论文解读

AI 想研究点子,前30个问题就耗尽了

让 AI 自己提出研究想法,它其实像个玩二十问的选手:前 30 个问题里几乎定下胜负,之后问再多也基本原地踏步。研究者把 200 篇 ICML 论文的「核心想法」藏起来,只给 AI 论文题目,让它靠问是/否问题去猜——结果发现,AI 不是不会判断:你给它一个好问题或好想法,它能认出来;但让它自己凭空想出来,它就卡住了。它失败的方式也很有规律:要么死磕一个方向不换,要么东一榔头西一棒子。这不是你明天能用的工具,但它给「自动做科研」的系统指了个方向:与其指望单个模型更会想,不如从外面多塞点候选想法进来,让它挑。

📄 原文摘要(英文)

Research ideation is iterative: researchers refine their ideas step by step based on feedback. Yet existing evaluations obscure this search process, making it difficult to understand how LLMs explore the space of research ideas and what limits their progress. We introduce Re-Search, which adapts the format of Twenty Questions to research ideation: given only the research problem of a published paper, a model must rediscover the paper's core idea by asking yes-or-no questions. Built from 200 ICML 2026 papers, Re-Search tracks how a model's idea evolves as it asks questions and receives binary oracle feedback, isolating ideation from execution. Across six recent LLMs, nearly all progress happens within the first 30 of 100 questions, which largely decide how the game ends; after that, most models plateau. Through controlled interventions on the candidate questions and the hypothesis space, we find that the bottleneck is generation, not judgment: models recognize a good idea or a good question when one is offered, but often cannot produce it on their own. We further analyze the ideation traces and find that models fail by either over-exploiting one direction or over-exploring too many. These findings point to concrete design principles for autoresearch systems: since models can judge candidates they cannot produce, enriching the search with externally sourced ideas and questions may be more effective than improving any single model's generation ability.

arXiv 原文

订阅 AI Pulse

每天 08:00 · 12:30 · 18:30 · 23:50 更新