从分布优化角度提升扩散模型对齐效果，理论保证更可靠。
Direct Distributional Optimization for Provable Alignment of Diffusion Models
- 直接优化概率分布，用对偶平均法求解
- 采样误差有端到端理论界，复杂度与等周条件无关
- 适用于RLHF、DPO等对齐任务，适合追求理论保障的研究者
我们提出一种基于分布优化的扩散模型对齐新方法，并提供严格的收敛性保证。将问题建模为概率分布上的正则化损失最小化，直接使用对偶平均法优化分布；通过Doob's h-变换近似得分函数，实现所学分布的采样。该框架具备严格收敛性保证和采样误差的端到端上界，表明当原始分布得分已知准确时，从偏移分布中采样的复杂度与等周条件无关。该方法可广泛应用于一般分布优化问题，包括强化学习中人类反馈（RLHF）、直接偏好优化（DPO）以及Kahneman-Tversky优化（KTO）。我们在合成数据和图像数据集上以DPO为目标进行了实证验证。
原文摘要 · Abstract (English)
We introduce a novel alignment method for diffusion models from distribution optimization perspectives while providing rigorous convergence guarantees. We first formulate the problem as a generic regularized loss minimization over probability distributions and directly optimize the distribution using the Dual Averaging method. Next, we enable sampling from the learned distribution by approximating its score function via Doob's $h$-transform technique. The proposed framework is supported by rigorous convergence guarantees and an end-to-end bound on the sampling error, which imply that when the original distribution's score is known accurately, the complexity of sampling from shifted distributions is independent of isoperimetric conditions. This framework is broadly applicable to general distribution optimization problems, including alignment tasks in Reinforcement Learning with Human Feedback (RLHF), Direct Preference Optimization (DPO), and Kahneman-Tversky Optimization (KTO). We empirically validate its performance on synthetic and image datasets using the DPO objective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。