让扩散模型生成图像时每一步都获得奖励，提升训练效率。
CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning
- 用余弦相似度动态分配每步的奖励，实现细粒度反馈。
- 在四个偏好奖励函数上样本效率提升1.25到2倍。
- 无需额外网络，适合需要高效微调的图像生成任务。
近期文本到图像（T2I）扩散模型微调采用强化学习（RL）对齐生成图像与可学习奖励函数。现有方法将去噪过程重构成马尔可夫决策过程以驱动优化，但存在奖励稀疏问题，仅在生成轨迹结束后获得单一延迟奖励，阻碍了对去噪步骤的精准归因，降低训练效率。为此，我们提出一种简单有效的信用分配框架，动态将密集奖励分配至各去噪步骤。具体而言，通过追踪中间图像与最终图像之间的余弦相似度变化，量化每一步对逐步缩小与目标图像距离的贡献。该方法避免使用额外辅助神经网络进行步骤级偏好建模，转而通过奖励塑造突出对图像质量影响更大的去噪阶段。实验表明，本方法在四个基于人类偏好的奖励函数下，样本效率提升1.25至2倍，且不损害原始最优策略性能。
原文摘要 · Abstract (English)
Recent advances in text-to-image (T2I) diffusion model fine-tuning leverage reinforcement learning (RL) to align generated images with learnable reward functions. The existing approaches reformulate denoising as a Markov decision process for RL-driven optimization. However, they suffer from reward sparsity, receiving only a single delayed reward per generated trajectory. This flaw hinders precise step-level attribution of denoising actions, undermines training efficiency. To address this, we propose a simple yet effective credit assignment framework that dynamically distributes dense rewards across denoising steps. Specifically, we track changes in cosine similarity between intermediate and final images to quantify each step's contribution on progressively reducing the distance to the final image. Our approach avoids additional auxiliary neural networks for step-level preference modeling and instead uses reward shaping to highlight denoising phases that have a greater impact on image quality. Our method achieves 1.25 to 2 times higher sample efficiency and better generalization across four human preference reward functions, without compromising the original optimal policy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。