arXiv:2604.12652cs.CVcs.AI2026-04被引 2

无需标注和训练,用预训练模型直接生成图像文本对齐奖励。

PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning

论文配图:PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning
图 1 · 摘自论文原文
  • 利用冻结的视觉语言模型计算令牌级交叉熵作为奖励信号。
  • 在多个基准上提升显著,最高净胜率提升26.8个百分点。
  • 适合希望低成本优化文生图模型对提示理解能力的研究者。

强化学习可提升文生图模型对提示词的遵循能力,但高质量奖励信号获取困难:CLIP分数过于粗略,基于视觉语言模型的奖励模型(如RewardDance)需昂贵的人工标注偏好数据并额外微调。本文提出PromptEcho,一种无需标注、无需奖励模型训练的奖励构建方法。给定生成图像和引导查询,PromptEcho以原始提示为标签,计算冻结的VLM在令牌级别的交叉熵损失,直接提取其预训练中编码的图文对齐知识。该奖励确定性高、计算高效,且随更强的开源VLM自动提升。为评估,我们构建了DenseAlignBench,一个包含丰富概念的密集描述基准,用于严格测试提示遵循能力。在两个主流文生图模型(Z-Image与QwenImage-2512)上的实验表明,PromptEcho在DenseAlignBench上分别实现+26.8pp与+16.2pp的净胜率提升,并在GenEval、DPG-Bench与TIIFBench上获得一致增益,无需任务特定训练。消融实验证实,PromptEcho全面优于同VLM的推理评分方法,且奖励质量随VLM规模增长。相关模型与基准将开源。

原文摘要 · Abstract (English)

Reinforcement learning (RL) can improve the prompt following capability of text-to-image (T2I) models, yet obtaining high-quality reward signals remains challenging: CLIP Score is too coarse-grained, while VLM-based reward models (e.g., RewardDance) require costly human-annotated preference data and additional fine-tuning. We propose PromptEcho, a reward construction method that requires \emph{no} annotation and \emph{no} reward model training. Given a generated image and a guiding query, PromptEcho computes the token-level cross-entropy loss of a frozen VLM with the original prompt as the label, directly extracting the image-text alignment knowledge encoded during VLM pretraining. The reward is deterministic, computationally efficient, and improves automatically as stronger open-source VLMs become available. For evaluation, we develop DenseAlignBench, a benchmark of concept-rich dense captions for rigorously testing prompt following capability. Experimental results on two state-of-the-art T2I models (Z-Image and QwenImage-2512) demonstrate that PromptEcho achieves substantial improvements on DenseAlignBench (+26.8pp / +16.2pp net win rate), along with consistent gains on GenEval, DPG-Bench, and TIIFBench without any task-specific training. Ablation studies confirm that PromptEcho comprehensively outperforms inference-based scoring with the same VLM, and that reward quality scales with VLM size. We will open-source the trained models and the DenseAlignBench.

文生图强化学习奖励设计视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。