用连续时间强化学习优化扩散模型，让生成结果更贴合提示词。
Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning
- 将得分匹配视为连续控制动作，构建连续时间强化学习框架。
- 在Stable Diffusion v1.5上实现更精准的文本到图像生成，提升对齐效果。
- 适合需要高精度提示对齐的生成模型微调场景，如艺术创作与设计。
基于人类反馈的强化学习（RLHF）已成为使扩散模型与输入提示对齐的关键步骤，以构建可靠的生成式AI模型。现有方法多采用离散时间形式，易引入离散化误差，且不适用于高阶或黑箱求解器的模型。本文提出一种基于连续时间强化学习的规范性微调方法，将目标建模为一个随机控制问题，通过奖励函数引导终态（终端状态）与输入提示对齐。核心思想是将得分匹配视为控制动作，从而建立与连续时间强化学习中策略优化和正则化的联系。为此，我们构建了新的连续时间强化学习策略优化框架，并利用扩散模型的结构特性拓展价值网络的设计空间。实验验证了该方法在大规模Stable Diffusion v1.5文本到图像任务中的优势，显著提升了生成结果与提示的契合度。
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF), which aligns a diffusion model with input prompt, has become a crucial step in building reliable generative AI models. Most works in this area use a discrete-time formulation, which is prone to induced discretization errors, and often not applicable to models with higher-order/black-box solvers. The objective of this study is to develop a disciplined approach to fine-tune diffusion models using continuous-time RL, formulated as a stochastic control problem with a reward function that aligns the end result (terminal state) with input prompt. The key idea is to treat score matching as controls or actions, and thereby making connections to policy optimization and regularization in continuous-time RL. To carry out this idea, we lay out a new policy optimization framework for continuous-time RL, and illustrate its potential in enhancing the value networks design space via leveraging the structural property of diffusion models. We validate the advantages of our method by experiments in downstream tasks of fine-tuning large-scale Text2Image models of Stable Diffusion v1.5.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。