用预训练多模态模型直接做图像生成的奖励函数，无需额外训练。
Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation
- 通过反向还原提示词评估生成图像质量，利用模型自身对齐能力。
- 在多个扩散模型和算法上显著提升生成效果，优于现有方法。
- 适合希望零样本构建奖励模型的研究者或开发者使用。
本文提出SpectraReward，一种无需训练的奖励函数，将预训练多模态大模型（MLLM）直接转化为图像生成强化学习中的即插即用奖励模型。不同于要求模型判断图像或回答分解问题，SpectraReward通过单次图像条件下的教师强制前向传播，测量原始提示词从生成图像中被恢复的程度，并以平均图像条件提示词对数似然作为奖励，直接复用模型已有的图文对齐能力，无需偏好标签或奖励模型微调。我们进一步提出Self-SpectraReward，一种适用于统一多模态模型的特例，其中策略的自身理解分支作为其生成分支的奖励模型，形成无需外部奖励模型或知识的闭环自优化框架。大量实验验证了SpectraReward的有效性：涵盖两种扩散模型、三种强化学习算法、九种来自四个MLLM家族的4B至235B参数的奖励模型骨干，以及五个分布外文本到图像基准。结果表明，SpectraReward与Self-SpectraReward均显著且一致地提升生成性能，优于先前基于MLLM的奖励训练方法。进一步分析显示，更大的奖励模型并非总更好，而Self-SpectraReward可媲美甚至超越更大规模的外部奖励模型，表明奖励-策略对齐是有效图像生成强化学习的关键因素。
原文摘要 · Abstract (English)
In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。