不依赖模型内部结构，直接优化噪声实现生成模型对齐。
Trust-Region Noise Search for Black-Box Alignment of Diffusion and Flow Models
- 用信任域搜索法只优化输入噪声，黑箱适配各类生成模型。
- 在图文生成、分子蛋白设计中显著提升生成质量。
- 无需微调模型，适合快速部署于不同任务场景。
在推理阶段对扩散模型和流模型进行对齐，通常依赖可微或低成本的奖励模型，或受限于预训练生成模型的结构，且存在内存与计算效率问题。本文提出一种基于信任域的噪声搜索算法（TRS），将生成模型和奖励模型视为黑箱，仅优化初始噪声。该方法在全局探索与局部优化间取得良好平衡，适用于多种生成设置和奖励模型，仅需极少超参数调整。我们在文本到图像、分子设计和蛋白质设计任务上评估了该方法，结果表明，其生成样本质量显著优于基线生成模型及其它优化噪声或完整反向采样轨迹的方法。代码已开源。
原文摘要 · Abstract (English)
Optimizing the noise samples of diffusion and flow models is an increasingly popular approach to align these models to target rewards at inference time. However, we observe that these approaches are usually restricted to differentiable or cheap reward models, the formulation of the underlying pretrained generative model, or are memory/compute inefficient. We instead propose a simple trust-region based search algorithm (TRS) which treats the pre-trained generative and reward models as a black-box and only optimizes the source noise. Our approach achieves a good balance between global exploration and local exploitation, and is versatile and easily adaptable to various generative settings and reward models with minimal hyperparameter tuning. We evaluate TRS across text-to-image, molecule and protein design tasks, and obtain significantly improved output samples over the base generative models and other inference-time alignment approaches which optimize the source noise sample, or even the entire reverse-time sampling noise trajectories in the case of diffusion models. Our source code is publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。