让少步扩散模型更精准对齐目标，提升图像生成质量。
Aligning Few-Step Diffusion Models with Dense Reward Difference Learning
- 设计双状态采样机制，实现每步密集奖励反馈
- 在少步场景下达成更高奖励对齐效果，优于现有方法
- 适合需要高效高质图像生成的落地应用
少步扩散模型能高效生成高分辨率图像，但因现有强化学习方法在低步数场景中状态空间有限、样本质量不佳，难以对齐特定下游目标。为此，我们提出步进式扩散策略优化（SDPO），专为少步扩散模型设计的新型强化学习框架。SDPO引入双状态轨迹采样机制，在每一步同时追踪噪声状态与预测的干净状态，提供密集奖励反馈，支持低方差、多步混合优化。为提升效率，进一步提出基于潜在相似性的密集奖励预测策略，减少昂贵的密集奖励查询。利用这些密集奖励，SDPO优化密集奖励差异学习目标，实现更频繁、更精细的策略更新。额外改进包括步进优势估计、时间重要性加权和步序打乱梯度更新，分别增强长期依赖建模、低步优先级处理与梯度稳定性。实验表明，SDPO在多种少步设置与任务中均持续取得更优的奖励对齐结果。代码已开源：https://github.com/ZiyiZhang27/sdpo。
原文摘要 · Abstract (English)
Few-step diffusion models enable efficient high-resolution image synthesis but struggle to align with specific downstream objectives due to limitations of existing reinforcement learning (RL) methods in low-step regimes with limited state spaces and suboptimal sample quality. To address this, we propose Stepwise Diffusion Policy Optimization (SDPO), a novel RL framework tailored for few-step diffusion models. SDPO introduces a dual-state trajectory sampling mechanism, tracking both noisy and predicted clean states at each step to provide dense reward feedback and enable low-variance, mixed-step optimization. For further efficiency, we develop a latent similarity-based dense reward prediction strategy to minimize costly dense reward queries. Leveraging these dense rewards, SDPO optimizes a dense reward difference learning objective that enables more frequent and granular policy updates. Additional refinements, including stepwise advantage estimates, temporal importance weighting, and step-shuffled gradient updates, further enhance long-term dependency, low-step priority, and gradient stability. Our experiments demonstrate that SDPO consistently delivers superior reward-aligned results across diverse few-step settings and tasks. Code is available at https://github.com/ZiyiZhang27/sdpo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。