用对比引导提升扩散模型对用户偏好的对齐效果
Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidance
- 将偏好对齐视为无分类器引导，用微调模型生成控制信号
- 通过正负样本分离训练，推理时相减得到更清晰的引导向量
- 无需重训练主模型，适用于Stable Diffusion系列，提升可控性
大规模文本到图像扩散模型与复杂人类偏好的对齐仍具挑战。尽管直接偏好优化（DPO）简单有效，但大规模微调常出现泛化差距。受测试时引导启发，我们提出将偏好对齐建模为无分类器引导（CFG）：微调的偏好模型作为采样过程中的外部控制信号。基于此，我们提出一种无需重训练基础模型的简单方法。为进一步提升泛化能力，我们将偏好学习解耦为两个模块，分别在正样本和负样本上训练，并在推理时通过相减（正向减负向）形成对比引导向量，按用户设定强度缩放后叠加至每步的基础预测。该方法获得更锐利且可控的对齐信号。我们在Stable Diffusion 1.5与Stable Diffusion XL上使用Pick-a-Pic v2与HPDv3评估，展现出一致的定量与定性提升。
原文摘要 · Abstract (English)
Aligning large-scale text-to-image diffusion models with nuanced human preferences remains challenging. While direct preference optimization (DPO) is simple and effective, large-scale finetuning often shows a generalization gap. We take inspiration from test-time guidance and cast preference alignment as classifier-free guidance (CFG): a finetuned preference model acts as an external control signal during sampling. Building on this view, we propose a simple method that improves alignment without retraining the base model. To further enhance generalization, we decouple preference learning into two modules trained on positive and negative data, respectively, and form a \emph{contrastive guidance} vector at inference by subtracting their predictions (positive minus negative), scaled by a user-chosen strength and added to the base prediction at each step. This yields a sharper and controllable alignment signal. We evaluate on Stable Diffusion 1.5 and Stable Diffusion XL with Pick-a-Pic v2 and HPDv3, showing consistent quantitative and qualitative gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。