提出Flow Density Control方法,让生成模型更灵活地优化复杂目标。
Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuning
- 将复杂优化问题分解为可解的简单微调步骤
- 支持风险规避、新颖性追求等非平均奖励目标
- 适合分子设计、图像生成等实际应用
将大规模生成模型适应于特定任务目标的同时保留先验知识,对分子设计、蛋白质对接和创意图像生成等实际应用至关重要。现有规范微调方法旨在最大化生成样本的期望奖励,并通过KL散度正则化保持预训练模型知识。本文解决更普遍的问题:优化超越平均奖励的一般效用,包括风险规避、新颖性追求、多样性度量及实验设计目标等;同时考虑超越KL散度的更一般先验信息保留方式,如最优传输距离和Renyi散度。为此,我们提出流密度控制(FDC)算法,将该复杂问题简化为一系列可通过现有可扩展方法求解的简单微调任务。在合理假设下,借助镜像流的最新理解,我们推导出所提方案的收敛性保证。我们在文本到图像和分子设计等示例场景中验证了方法,结果表明FDC能够引导预训练生成模型优化目标并解决当前微调方案无法触及的实际任务。
原文摘要 · Abstract (English)
Adapting large-scale foundation flow and diffusion generative models to optimize task-specific objectives while preserving prior information is crucial for real-world applications such as molecular design, protein docking, and creative image generation. Existing principled fine-tuning methods aim to maximize the expected reward of generated samples, while retaining knowledge from the pre-trained model via KL-divergence regularization. In this work, we tackle the significantly more general problem of optimizing general utilities beyond average rewards, including risk-averse and novelty-seeking reward maximization, diversity measures for exploration, and experiment design objectives among others. Likewise, we consider more general ways to preserve prior information beyond KL-divergence, such as optimal transport distances and Renyi divergences. To this end, we introduce Flow Density Control (FDC), a simple algorithm that reduces this complex problem to a specific sequence of simpler fine-tuning tasks, each solvable via scalable established methods. We derive convergence guarantees for the proposed scheme under realistic assumptions by leveraging recent understanding of mirror flows. Finally, we validate our method on illustrative settings, text-to-image, and molecular design tasks, showing that it can steer pre-trained generative models to optimize objectives and solve practically relevant tasks beyond the reach of current fine-tuning schemes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。