用梯度匹配提升生成模型对齐人类偏好的效率与保真度
Value Gradient Guidance for Flow Matching Alignment
- 基于最优控制理论，通过匹配价值函数梯度来调整速度场
- 在有限算力下实现快速微调，同时保持原始生成先验
- 适合需高效对齐奖励模型的生成模型优化场景
尽管已有方法可将流匹配模型——一种流行且高效的生成模型——与人类偏好对齐，但现有方法难以兼顾适应效率和概率上合理的先验保持。本文基于最优控制理论，提出VGG-Flow，一种基于梯度匹配的预训练流匹配模型微调方法。该算法的核心思想是：微调后速度场与预训练速度场之间的最优差异，应与价值函数的梯度场相匹配。该方法不仅利用了奖励模型的一阶信息，还通过价值函数的启发式初始化实现快速适应。实验表明，在主流文本到图像流匹配模型Stable Diffusion 3上，本方法能在有限计算预算下实现有效且先验保留的对齐。
原文摘要 · Abstract (English)
While methods exist for aligning flow matching models--a popular and effective class of generative models--with human preferences, existing approaches fail to achieve both adaptation efficiency and probabilistically sound prior preservation. In this work, we leverage the theory of optimal control and propose VGG-Flow, a gradient-matching-based method for finetuning pretrained flow matching models. The key idea behind this algorithm is that the optimal difference between the finetuned velocity field and the pretrained one should be matched with the gradient field of a value function. This method not only incorporates first-order information from the reward model but also benefits from heuristic initialization of the value function to enable fast adaptation. Empirically, we show on a popular text-to-image flow matching model, Stable Diffusion 3, that our method can finetune flow matching models under limited computational budgets while achieving effective and prior-preserving alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。