用改进的强化学习方法让扩散模型生成更自然且多样化的图像。
Diffusion Fine-Tuning via Reparameterized Policy Gradient of the Soft Q-Function
- 通过重参数化策略梯度优化软Q函数，避免奖励过拟合。
- 在文本到图像任务中同时提升目标奖励与样本多样性。
- 适合需要高质量生成结果的研究者和开发者使用。
扩散模型虽能生成高似然样本，但常需与下游目标对齐。现有微调方法易导致奖励过优化，产生高奖励却不自然的样本，降低多样性。为此，我们提出基于软Q函数的扩散微调（SQDF），一种带KL正则化的强化学习方法，采用无需训练、可微分的软Q函数估计的重参数化策略梯度。SQDF进一步引入三项创新：在去噪过程中使用折扣因子实现合理信用分配；融合一致性模型以优化Q函数估计；采用离线策略回放缓冲区提升模式覆盖并平衡奖励与多样性。实验表明，SQDF在文本到图像对齐任务中显著提升目标奖励，同时保持样本多样性。在在线黑箱优化中，也实现了高样本效率，且保持自然性与多样性。代码已开源：https://github.com/Shin-woocheol/SQDF。
原文摘要 · Abstract (English)
Diffusion models excel at generating high-likelihood samples but often require alignment with downstream objectives. Existing fine-tuning methods for diffusion models significantly suffer from reward over-optimization, resulting in high-reward but unnatural samples and degraded diversity. To mitigate over-optimization, we propose Soft Q-based Diffusion Finetuning (SQDF), a novel KL-regularized RL method for diffusion alignment that applies a reparameterized policy gradient of a training-free, differentiable estimation of the soft Q-function. SQDF is further enhanced with three innovations: a discount factor for proper credit assignment in the denoising process, the integration of consistency models to refine Q-function estimates, and the use of an off-policy replay buffer to improve mode coverage and manage the reward-diversity trade-off. Our experiments demonstrate that SQDF achieves superior target rewards while preserving diversity in text-to-image alignment. Furthermore, in online black-box optimization, SQDF attains high sample efficiency while maintaining naturalness and diversity. Our code is available at https://github.com/Shin-woocheol/SQDF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。