AI审稿人只学挑错,却学会了看合同
AI审稿人这件事,研究者先砍掉了「有趣」「清晰」这些主观标准,只教模型判断论文的论证站不站得住。做法是给 1000 篇 ICLR 投稿里的 2.3 万条审稿意见逐条打上质量标签,再用强化学习训练模型,奖励同时看两件事:说出的问题是否扎实(精确率),以及有没有覆盖到人类审稿人提到的高质量意见(召回率)。结果这个叫 PaperCritic 的模型在同类测试集上超过了前沿大模型和别的训练方法。真正意外的是迁移:只学过审 ICLR 论文,它去审 Nature 系列期刊照样能用,连代码审查、法律合同审查、小说写作反馈这些完全不同的挑错任务也沾上了光。它不是你明天能用上的工具,但它说明「判断一段论证是否站得住」可能是一种可以跨领域搬家的通用能力。
📄 原文摘要(英文)
As AI conference submissions grow rapidly, AI reviewers are being considered as an alternative. However, simply imitating human reviewing is problematic because reviewing serves multiple purposes, including evaluating papers' soundness, interestingness, and clarity, the last two of which are subjective. In this work, we focus solely on evaluating soundness, the most important and objective criterion, and define the learning objective as jointly maximizing (1) precision, based on the soundness and substantiveness of review points, and (2) recall, the coverage of high-quality reference review points. Under this problem formulation, we construct a quality-labeled review point dataset, PaperSound-23K, using 1K ICLR submissions and 23K human-written and LLM-generated review points. Using this dataset, we train an AI reviewer model, PaperCritic, through reinforcement learning with the F1 score of precision and recall as the reward. Our experiments show that PaperCritic outperforms frontier models and models trained with other AI reviewer training methods on an in-domain test set. Surprisingly, learning to review ICLR submissions also transfers to reviewing Nature-family papers and even to out-of-domain reviewing tasks (e.g., code review, legal contract review, and feedback on story writing).