AI 自己造考试题，能考出真水平吗？
让 AI 自己给自己出题、自己判卷，听起来是省事又高效，但新研究 EvalResearchBench 泼了盆冷水：AI 造出的“考试”和人类专家设计的标准考试，在给模型排名时只有约 75% 的一致度，远低于人类之间的 91%。更尴尬的是，没有一个 AI 在所有指标上领先，而且最会出题的 AI 在“密封题”(它没见过的测试)上反而表现平平。AI 确实能通过试测修修补补，但它的考卷还是会漏判答案、烧光预算，或者让几道题就决定整个分数。这不是你明天能用的工具，但它提醒我们：AI 的“自我改进”目前还卡在“自己考自己”这一步——它连一套靠谱的考卷都还造不出来。
📄 原文摘要(英文)
Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. The best evaluators order about 75\% of candidate pairs as the targets do, below the 91\% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement. Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.