用采样方法对齐少步生成模型，无需复杂假设。
Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inference
- 通过采样推断优化生成器参数，仅需访问生成器和参考分布。
- 在机器人操作和图像生成任务中均超越现有基线，提升显著。
- 适用于多种生成模型，适合研究生成对齐与强化学习者。
对少步生成模型进行对齐极具挑战性，因现有对齐框架通常依赖于严格假设：可计算似然、特定的ODE/SDE求解器或特定模型族。本文提出FAV（Few-step Generative Models Alignment via Sample-based Variational Inference），一种仅需生成器和参考分布采样访问的通用对齐框架。将对齐问题建模为从以参考分布为锚点的奖励倾斜分布中采样。利用Stein变分梯度下降作为基于样本的变分推断方案，并通过固定点回归将粒子更新摊销至生成器参数。在机器人操控与图像生成两个领域进行评估：在离线及离线到在线强化学习任务中，对机器人策略对齐的表现优于现有基线，在56个离线任务和30个混合任务中均实现提升；在图像生成对齐方面，成功微调多种少步骨干模型（包括GAN、漂移模型、一致性模型、流映射），支持从ImageNet-256到1024²的文本到图像生成。代码已开源。
原文摘要 · Abstract (English)
Aligning a few-step generative model is challenging, since existing alignment frameworks typically rely on restrictive assumptions: a tractable likelihood, a specific ODE/SDE solver, or a particular model family. We introduce FAV, Few-step Generative Models Alignment via Sample-based Variational Inference, a general alignment framework that requires only sample access to the generator and the reference distribution. We cast alignment as sampling from a reward-tilted distribution anchored to a reference distribution. We leverage Stein Variational Gradient Descent as a sample-based variational inference scheme and amortize its particle updates into the generator parameters via fixed-point regression. We evaluate FAV on two domains: robotics manipulation and image generator alignment. On generative policy alignment for robotic manipulation, FAV outperforms prevailing policy extraction baselines across 56 offline and 30 offline-to-online RL tasks. For image generator alignment, FAV fine-tunes diverse few-step backbones, including GAN, drifting model, consistency models, and flow maps, scaling from ImageNet-$256$ to 1024$^2$ text-to-image synthesis. Code is available at https://github.com/Jaewoopudding/FAV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。