提出新型微调方法,让扩散模型更高效精准地生成图像视频。
Half-order Fine-Tuning for Diffusion Model: A Recursive Likelihood Ratio Optimizer
- 通过递归似然比优化器,实现无偏低方差梯度估计。
- 在图像与视频生成任务中显著提升性能,避免训练失败。
- 适合需要高质量生成的视觉应用开发者使用。
概率扩散模型(DM)通过递归链结构进行内容生成,已成为视觉生成的强大框架。预训练后,需对模型进行有效对齐以满足下游应用需求。现有方法多基于强化学习(RL)或截断反向传播(BP),但前者样本效率低,后者存在梯度估计偏差,导致改进有限甚至训练失败。为此,本文提出递归似然比(RLR)优化器,一种针对扩散模型的半阶(HO)微调范式。该方法通过重构递归扩散链中的计算图,使梯度估计无偏且方差更低。我们理论分析了方法的偏差、方差与收敛性。大量实验验证了其在图像与视频生成任务上的优越性。此外,我们设计了一种与RLR自然契合的新提示技术,实现协同增效。
原文摘要 · Abstract (English)
The probabilistic diffusion model (DM), generating content by inferencing through a recursive chain structure, has emerged as a powerful framework for visual generation. After pre-training on enormous data, the model needs to be properly aligned to meet requirements for downstream applications. How to efficiently align the foundation DM is a crucial task. Contemporary methods are either based on Reinforcement Learning (RL) or truncated Backpropagation (BP). However, RL and truncated BP suffer from low sample efficiency and biased gradient estimation, respectively, resulting in limited improvement or, even worse, complete training failure. To overcome the challenges, we propose the Recursive Likelihood Ratio (RLR) optimizer, a Half-Order (HO) fine-tuning paradigm for DM. The HO gradient estimator enables the computation graph rearrangement within the recursive diffusive chain, making the RLR's gradient estimator an unbiased one with lower variance than other methods. We theoretically investigate the bias, variance, and convergence of our method. Extensive experiments are conducted on image and video generation to validate the superiority of the RLR. Furthermore, we propose a novel prompt technique that is natural for the RLR to achieve a synergistic effect.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。