arXiv:2606.17979cs.AI2026-06

让文本生成的强化学习更精准,按图像关键区域动态分配奖励。

STAR: SpatioTemporal Adaptive Reward Allocation for Text-to-Image RL Post-Training

论文配图:STAR: SpatioTemporal Adaptive Reward Allocation for Text-to-Image RL Post-Training
图 1 · 摘自论文原文
  • 根据文本注意力定位图像关键区域,动态分配奖励
  • 在三个任务上分别提升至0.9759、0.9757和23.60
  • 无需改外部奖励,适合优化生成质量的工程师

现有的文本到图像生成强化学习后训练方法通常将最终图像奖励转化为单一标量优势,并以相同强度应用于整个生成轨迹。然而,文本到图像生成具有天然的时间与空间结构:不同去噪步骤负责不同生成阶段,真正决定文本对齐的内容往往仅出现在图像局部。这种粒度不匹配导致策略更新难以聚焦于真正影响奖励的生成组件。为此,我们提出 extbf{SpatioTemporal Adaptive Reward (STAR) Allocation},用于文本到图像扩散与流模型的强化学习后训练。STAR利用生成模型内部的文本-图像注意力,从用户提示中真正关心的核心内容出发,构建随去噪步骤和采样轨迹动态变化的空间分配图,将相同的组内相对优势分配给更相关的潜在区域,几乎无额外计算开销。随后,通过空间分辨的策略目标对这些区域施加强化更新。以Stable Diffusion 3.5 Medium为基线模型,在GenEval、OCR文本渲染和PickScore三个任务上评估,实验结果表明,STAR在不改变外部奖励源的前提下,显著提升了组合语义对齐、文本渲染和偏好优化能力,分别达到0.9759、0.9757和23.60。

原文摘要 · Abstract (English)

Existing RL post-training methods for text-to-image generation usually convert the final-image reward into a single scalar advantage and apply it with the same strength to the entire generative trajectory. However, text-to-image generation naturally has temporal and spatial structure: different denoising steps are responsible for different generation stages, and the content that truly determines text alignment often appears only in part of the image. This granularity mismatch makes it difficult for policy updates to focus on the generative components that actually affect the reward. To address this issue, we propose \textbf{SpatioTemporal Adaptive Reward (STAR) Allocation} for RL post-training of text-to-image diffusion and flow models. STAR uses text-image attention inside the generative model and starts from the core content that the user truly cares about in the prompt. It constructs spatial allocation maps that dynamically vary across denoising steps and rollouts, and allocates the same group-relative advantage to more relevant latent regions with almost no additional computational overhead. STAR then applies stronger policy updates to these regions through a spatially resolved policy objective. We use Stable Diffusion 3.5 Medium as the base model and evaluate on three tasks: GenEval, OCR text rendering, and PickScore. Experimental results show that STAR improves compositional semantic alignment, text rendering, and preference optimization without changing the external reward source, achieving $\mathbf{0.9759}$, $\mathbf{0.9757}$, and $\mathbf{23.60}$ on GenEval, OCR, and PickScore, respectively.

文本生成强化学习图像生成扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。