让AI自己出题考自己：把开放任务变成可验证游戏
现在的AI强化学习有个死穴：数学和代码题能自动判对错，所以能越练越强；但写文章、创意写作这类开放任务，只能靠人类打分或另一个AI当裁判，又慢又可能偏心。这篇论文的思路很反直觉：不找更好的裁判，而是把任务本身改造成一个“有标准答案”的游戏。他们设计了一个类似“谁是卧底”的多智能体环境：几个AI拿到不对称信息，共同完成同一个任务，然后投票找出卧底。因为卧底身份是预设的，投票结果天然可验证，而“能不能找出卧底”又和任务质量强相关——写得好才容易暴露别人。实验显示，这种“任务转化”让AI在总结、创意写作上超过了现有自我改进方法，在数学推理上也有提升。它不是你明天就能用的工具，但指向一个趋势：与其纠结怎么给开放任务打分，不如把任务设计成自带分数的游戏。
📄 原文摘要(英文)
Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs) by enabling large-scale optimization. However, its applicability remains largely limited to domains such as mathematics and coding, where correctness can be deterministically verified. Open-ended tasks instead often rely on human preferences, reward models, or LLM-based judges, introducing evaluation bias, judge capability bottlenecks, and additional inference costs.Drawing on the principle of self-supervised learning, which constructs pretext tasks to derive supervision from the data itself, we propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks. RLSVR transforms open-ended tasks into verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals. We instantiate RLSVR with SpyRL, a multi-agent self-play environment inspired by Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Because the spy identity is predetermined, voting outcomes provide fully verifiable rewards, while successful identification remains closely related to output quality. Experiments on text summarization, creative writing, and mathematical reasoning show that SpyRL outperforms existing self-improvement methods on non-verifiable tasks and yields consistent gains on verifiable reasoning tasks. These results demonstrate that task transformation can extend scalable RLVR-based self-improvement beyond inherently verifiable domains. Models and code have been released at https://github.com/wangqinsi1/SpyRL.