用扩散模型自身处理噪声潜空间，实现更高效精准的图像偏好优化
Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization
- 直接在含噪潜空间中构建奖励模型，利用扩散模型天然处理多噪声层级的能力
- 新方法在多个偏好任务上显著提升效果，训练速度比现有方法快2.5至28倍
- 适合追求高效高质图像生成对齐的研究者与开发者
扩散模型的偏好优化旨在使图像生成结果符合人类偏好。以往方法通常使用视觉语言模型（VLM）作为像素级奖励模型来近似人类偏好，但在进行步骤级偏好优化时，难以处理不同时间步的含噪图像，且需复杂转换至像素空间。本文表明，预训练的扩散模型天然适用于在含噪潜空间中进行步骤级奖励建模，因其专为处理不同噪声水平的潜空间图像而设计。为此，我们提出潜空间奖励模型（LRM），复用扩散模型组件以预测任意时间步潜空间图像的偏好。基于LRM，我们进一步提出潜空间偏好优化（LPO），直接在含噪潜空间中执行步骤级偏好优化。实验表明，LPO显著提升了模型在通用性、美学及文本-图像对齐偏好上的表现，同时相比现有方法训练速度提升2.5至28倍。代码与模型已开源。
原文摘要 · Abstract (English)
Preference optimization for diffusion models aims to align them with human preferences for images. Previous methods typically use Vision-Language Models (VLMs) as pixel-level reward models to approximate human preferences. However, when used for step-level preference optimization, these models face challenges in handling noisy images of different timesteps and require complex transformations into pixel space. In this work, we show that pre-trained diffusion models are naturally suited for step-level reward modeling in the noisy latent space, as they are explicitly designed to process latent images at various noise levels. Accordingly, we propose the Latent Reward Model (LRM), which repurposes components of the diffusion model to predict preferences of latent images at arbitrary timesteps. Building on LRM, we introduce Latent Preference Optimization (LPO), a step-level preference optimization method conducted directly in the noisy latent space. Experimental results indicate that LPO significantly improves the model's alignment with general, aesthetic, and text-image alignment preferences, while achieving a 2.5-28x training speedup over existing preference optimization methods. Our code and models are available at https://github.com/Kwai-Kolors/LPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。