让AI自己当裁判：不用训练就能打分
训练AI画图通常需要人类给每张图打分，成本高且主观。这篇论文发现，多模态大模型（如能看懂图文的模型）天生就能当裁判：只要让模型看生成的图，再让它“读”出原本的提示词，读得越准说明图越好。这个方法不需要任何额外训练，直接拿现成模型用就行。更妙的是，同一个模型可以既当画家又当裁判——自己画完自己打分，形成自我改进的闭环。实验证明，这种自评自改的方式甚至比用更大模型当裁判效果更好。它不是你明天就能直接用的工具，但揭示了AI自我评估的新思路：与其依赖外部标准，不如让AI自己判断是否忠实于指令。
📄 原文摘要(英文)
In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/