arXiv:2605.15855cs.CV2026-05被引 7

只在生成关键阶段优化,效率和效果双提升。

Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?

论文配图:Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?
图 1 · 摘自论文原文
  • 按生成阶段动态决定是否用强化学习干预
  • 性能提升66%,计算成本降低59%
  • 适合追求高效高质图像生成的研究者

尽管扩散模型生成图像表现优异,但其重建目标限制了与人类偏好的对齐。强化学习可通过显式奖励实现对齐,但现有方法通常在完整去噪轨迹上应用强化学习,导致计算成本高且偏好对齐效果弱——做得多却成效低。我们观察到强化学习在不同去噪阶段的影响差异显著:早期图像结构不稳定,远离最终奖励信号,此时应用强化学习会导致奖励延迟和动作-奖励错配,引发高方差和更新效率低下;后期奖励增益趋于饱和,持续训练易过拟合局部细节,加剧奖励劫持。为此,我们提出AdaScope,一种增强型插件式强化学习方案,通过感知去噪过程中的结构演变与语义一致性,自适应识别最优干预时机,并在去噪收敛、奖励增益饱和时动态终止训练。结果实现罕见的双重收益:生成质量显著提升,计算成本大幅降低。我们为设计提供理论依据。相比最先进方法,AdaScope性能提升66%,计算成本降低59%。

原文摘要 · Abstract (English)

Despite strong image-generation performance, diffusion models' reconstruction objectives limit alignment with human preferences. RL enables such alignment through explicit rewards. However, most studies apply RL to the full denoising trajectory, making it computationally costly and weakening preference alignment, i.e., doing more but achieving less. We observe that the impact of RL fine-tuning varies significantly across denoising stages. In the early stage, image structures are unstable and distant from the final reward signal. Applying RL at this stage leads to delayed rewards and action-reward mismatching, resulting in high variance and inefficient updates. Conversely, in the later stage, reward gains saturate, and continued training tends to overfit local details, intensifying reward hacking. To tackle these challenges, we propose AdaScope, an RL-enhanced plug-in that improves generation quality while reducing computational cost. Specifically, AdaScope adaptively identifies the optimal intervention timing for RL by perceiving the structural evolution and semantic consistency during denoising, and dynamically terminates training once the denoising converges and reward gains saturate. As a result, it achieves a rare 'dual benefit': a reduction in computational costs alongside a significant performance improvement. We offer theoretical grounds for the design of AdaScope. Compared with state-of-the-art methods, AdaScope improves performance by 66% while cutting computational cost by 59%.

扩散模型强化学习效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。