让AI自己当裁判：不用训练，看图就能打分
训练AI画图通常需要人类打分或额外训练一个裁判模型，费时费力。这篇论文发现，现有的多模态大模型（比如能看图说话的AI）天生就能当裁判：只要让模型看一眼生成的图片，再让它“复述”出原本的提示词，复述得越准，说明图片越好。这个方法不需要任何额外训练，直接拿来用就行。实验证明，用这个打分方法去优化AI画图，效果比之前需要训练裁判的方法还好，而且模型越大不一定越准，关键是裁判和画图AI要“合拍”。虽然你明天用不上，但它让AI自我改进变得更简单——以后AI画图可能不需要人类反复调教了。
📄 原文摘要(英文)
In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/