通过分层强化学习提升扩散模型微调稳定性，避免模式崩溃。
Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning
- 分阶段动态微调扩散模型，每轮迭代逐步优化性能
- 仅对关键去噪步骤微调，显著减少模式崩溃现象
- 适合需高多样性输出的生成任务研究者参考
通过强化学习（RL）微调基础扩散模型（DMs）在对齐下游目标方面表现良好，但存在训练不稳定和模式崩溃等问题。本文提出分层奖励微调（HRF）方法，利用扩散模型的层级结构，在每个训练周期中动态调整，实现持续评估与逐步优化。研究发现，并非所有去噪步骤都需要微调。因此，除剪裁外，还采用滑动窗口机制在不同学习阶段对模型参数进行正则化。在去噪扩散策略优化方法上的验证表明，使用HRF训练的模型在保持下游任务多样性方面表现更优，同时不牺牲平均奖励，提升了微调鲁棒性。
原文摘要 · Abstract (English)
Fine-tuning foundation models via reinforcement learning (RL) has proven promising for aligning to downstream objectives. In the case of diffusion models (DMs), though RL training improves alignment from early timesteps, critical issues such as training instability and mode collapse arise. We address these drawbacks by exploiting the hierarchical nature of DMs: we train them dynamically at each epoch with a tailored RL method, allowing for continual evaluation and step-by-step refinement of the model performance (or alignment). Furthermore, we find that not every denoising step needs to be fine-tuned to align DMs to downstream tasks. Consequently, in addition to clipping, we regularise model parameters at distinct learning phases via a sliding-window approach. Our approach, termed Hierarchical Reward Fine-tuning (HRF), is validated on the Denoising Diffusion Policy Optimisation method, where we show that models trained with HRF achieve better preservation of diversity in downstream tasks, thus enhancing the fine-tuning robustness and at uncompromising mean rewards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。