无需奖励模型，用反演技术让扩散模型更精准高效对齐人类偏好。
Inversion-DPO: Precise and Efficient Post-Training for Diffusion Models
- 用DDIM反演替代难采样的后验分布，避免训练奖励模型。
- 在图文生成和组合图像生成任务中显著提升质量与效率。
- 适合追求高精度、低算力消耗的扩散模型优化场景。
扩散模型（DMs）的最新进展依赖于后训练对齐方法以更好符合人类偏好。然而，这些方法通常需要计算量巨大的基础模型与奖励模型训练，不仅带来巨大计算开销，还可能影响模型准确性和训练效率。为此，我们提出Inversion-DPO，一种新颖的对齐框架，通过将直接偏好优化（DPO）与DDIM反演结合，重构扩散模型的对齐过程。该方法利用获胜与失败样本到噪声的确定性反演，解决扩散型DPO中不可行的后验采样问题，从而构建新的后训练范式。该范式无需辅助奖励模型或近似手段，显著提升训练的精确性与效率。我们在文本到图像生成和组合图像生成两个任务上验证了该方法，实验表明其性能显著优于现有方法，并能生成高保真、结构一致的组合图像。针对组合图像生成任务，我们构建了一个包含11,140张带复杂结构标注与综合评分的配对数据集，以增强生成模型的组合能力。Inversion-DPO为扩散模型的高效高精度对齐开辟新路径，推动其在复杂真实生成任务中的应用。代码已开源。
原文摘要 · Abstract (English)
Recent advancements in diffusion models (DMs) have been propelled by alignment methods that post-train models to better conform to human preferences. However, these approaches typically require computation-intensive training of a base model and a reward model, which not only incurs substantial computational overhead but may also compromise model accuracy and training efficiency. To address these limitations, we propose Inversion-DPO, a novel alignment framework that circumvents reward modeling by reformulating Direct Preference Optimization (DPO) with DDIM inversion for DMs. Our method conducts intractable posterior sampling in Diffusion-DPO with the deterministic inversion from winning and losing samples to noise and thus derive a new post-training paradigm. This paradigm eliminates the need for auxiliary reward models or inaccurate appromixation, significantly enhancing both precision and efficiency of training. We apply Inversion-DPO to a basic task of text-to-image generation and a challenging task of compositional image generation. Extensive experiments show substantial performance improvements achieved by Inversion-DPO compared to existing post-training methods and highlight the ability of the trained generative models to generate high-fidelity compositionally coherent images. For the post-training of compostitional image geneation, we curate a paired dataset consisting of 11,140 images with complex structural annotations and comprehensive scores, designed to enhance the compositional capabilities of generative models. Inversion-DPO explores a new avenue for efficient, high-precision alignment in diffusion models, advancing their applicability to complex realistic generation tasks. Our code is available at https://github.com/MIGHTYEZ/Inversion-DPO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。