通过最优控制框架提升流模型对齐效果，训练更稳定高效。
Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipeline
- 将偏好对齐建模为速度场的最优控制问题，直接回归目标控制信号。
- 截断伴随方案聚焦轨迹末端，计算量减少但对齐质量保持不变。
- 支持灵活权衡对齐强度与分布保真度，适合高阶生成任务研究者。
我们提出一种确定性伴随匹配框架，将基于流的生成模型的人类偏好对齐问题建模为速度场上的最优控制问题。可在当前策略下直接回归由值梯度诱导的目标控制，实现简单且稳定的训练目标。在此基础上，引入截断伴随方案，仅关注轨迹末端部分，该区域集中了奖励相关信号，从而大幅降低计算开销，同时保持对齐质量。此外，该框架超越标准KL正则化，允许在对齐强度与分布保真度之间实现更灵活的权衡。在SiT-XL/2和FLUX.2-Klein-4B上的实验表明，多个对齐指标均有持续提升，多样性与模式保真度也显著改善。
原文摘要 · Abstract (English)
We propose a deterministic adjoint matching framework that formulates human preference alignment for flow-based generative models as an optimal control problem over velocity fields. One can directly regress the control toward a value-gradient-induced target under the current policy, leading to a simple and stable training objective. Building on this perspective, we introduce a truncated adjoint scheme that focuses computation on the terminal portion of the trajectory, where reward-relevant signals concentrate, which yields substantial computational savings while preserving alignment quality. We further generalize the framework beyond standard KL-based regularization, allowing more flexible trade-offs between alignment strength and distributional preservation. Experiments on SiT-XL/2 and FLUX.2-Klein-4B demonstrate consistent gains across multiple alignment metrics, along with substantially improved diversity and mode preservation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。