无需重训练，用噪声调控让图像生成更符合用户偏好。
Training-free Diffusion Model Alignment with Sampling Demons
- 通过控制去噪过程中的噪声分布，实现推理时的偏好对齐。
- 在文本到图像生成中显著提升美学评分，支持不可导奖励函数。
- 适用于视觉语言模型和人工评估等非可导奖励场景，易集成。
将扩散模型与用户偏好对齐一直是关键挑战。现有方法要么需要重训练，要么仅限于可微分的奖励函数。为此，我们提出一种称为Demon的随机优化方法，在推理阶段引导去噪过程，无需反向传播或模型重训练。该方法通过在去噪步骤中控制噪声分布，使密度集中在高奖励区域。我们提供了全面的理论和实证证据支持该方法，包括使用不可导奖励源（如视觉-语言模型API和人类判断）的实验。据我们所知，这是首个无反向传播、推理时的扩散模型偏好对齐方法。该方法可轻松集成到现有扩散模型中，无需额外训练。实验表明，该方法显著提升了文本到图像生成的平均美学评分。代码已开源：https://github.com/aiiu-lab/DemonSampling。
原文摘要 · Abstract (English)
Aligning diffusion models with user preferences has been a key challenge. Existing methods for aligning diffusion models either require retraining or are limited to differentiable reward functions. To address these limitations, we propose a stochastic optimization approach, dubbed Demon, to guide the denoising process at inference time without backpropagation through reward functions or model retraining. Our approach works by controlling noise distribution in denoising steps to concentrate density on regions corresponding to high rewards through stochastic optimization. We provide comprehensive theoretical and empirical evidence to support and validate our approach, including experiments that use non-differentiable sources of rewards such as Visual-Language Model (VLM) APIs and human judgements. To the best of our knowledge, the proposed approach is the first inference-time, backpropagation-free preference alignment method for diffusion models. Our method can be easily integrated with existing diffusion models without further training. Our experiments show that the proposed approach significantly improves the average aesthetics scores for text-to-image generation. Implementation is available at https://github.com/aiiu-lab/DemonSampling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。