用变分期望最大化框架优化扩散模型，兼顾奖励与多样性。
Diffusion Alignment as Variational Expectation-Maximization
- 将扩散对齐分为搜索与更新两步迭代进行
- 在文本到图像和DNA设计中同时提升奖励与样本多样性
- 适合需要高质量多样输出的生成任务
扩散对齐旨在优化扩散模型以适配下游目标。现有基于强化学习或直接反向传播的方法虽在最大化奖励方面取得显著成效，但常面临奖励过优化和模式崩溃问题。我们提出扩散对齐的变分期望最大化框架（DAV），将扩散对齐建模为交替执行的E步和M步迭代过程：E步利用测试时搜索生成多样且高奖励的样本；M步则使用E步发现的样本优化扩散模型。实验表明，DAV可在连续与离散任务中同时优化奖励并保持多样性，涵盖文本到图像生成与DNA序列设计。代码已公开于 https://github.com/Jaewoopudding/dav。
原文摘要 · Abstract (English)
Diffusion alignment aims to optimize diffusion models for the downstream objective. While existing methods based on reinforcement learning or direct backpropagation achieve considerable success in maximizing rewards, they often suffer from reward over-optimization and mode collapse. We introduce Diffusion Alignment as Variational Expectation-Maximization (DAV), a framework that formulates diffusion alignment as an iterative process alternating between two complementary phases: the E-step and the M-step. In the E-step, we employ test-time search to generate diverse and reward-aligned samples. In the M-step, we refine the diffusion model using samples discovered by the E-step. We demonstrate that DAV can optimize reward while preserving diversity for both continuous and discrete tasks: text-to-image synthesis and DNA sequence design. Our code is available at https://github.com/Jaewoopudding/dav.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。