arXiv:2605.11480cs.LG2026-05被引 1

提出高效方法EAM,让扩散模型对齐人类偏好更快更省资源。

Efficient Adjoint Matching for Fine-tuning Diffusion Models

论文配图:Efficient Adjoint Matching for Fine-tuning Diffusion Models
图 1 · 摘自论文原文
  • 用线性基漂移重构最优控制问题,简化生成过程
  • 训练速度比原方法快4倍,性能不降反升
  • 适合需要快速微调扩散模型的开发者和研究者

奖励微调已成为对齐预训练扩散模型与流模型与人类偏好的常用方法。在基于奖励梯度的方法中,伴随匹配(AM)通过将奖励微调建模为随机最优控制(SOC)问题,提供了严谨的框架。然而,AM不可避免地带来巨大计算开销:(i)在无记忆动力学下需模拟完整生成轨迹,导致大量函数求值;(ii)需沿每条采样轨迹进行后向常微分方程(ODE)的伴随状态仿真。本文观察到这两个瓶颈均与预训练模型继承的非平凡基漂移密切相关。受此启发,我们提出高效伴随匹配(EAM),通过引入线性基漂移并相应修改终端代价,重构了该SOC问题。这一重构消除了双重效率瓶颈:支持训练时采用少步数确定性ODE求解器进行采样,并获得闭式伴随解,从而无需后向伴随仿真。在标准文本到图像奖励微调基准上,EAM收敛速度最高可达AM的4倍,且在PickScore、ImageReward、HPSv2.1、CLIPScore和Aesthetics等指标上达到或超越AM表现。

原文摘要 · Abstract (English)

Reward fine-tuning has become a common approach for aligning pretrained diffusion and flow models with human preferences in text-to-image generation. Among reward-gradient-based methods, Adjoint Matching (AM) provides a principled formulation by casting reward fine-tuning as a stochastic optimal control (SOC) problem. However, AM inevitably requires a substantial computational cost: it requires (i) stochastic simulation of full generative trajectories under memoryless dynamics, resulting in a large number of function evaluations, and (ii) backward ODE simulation of the adjoint state along each sampled trajectory. In this work, we observe that both bottlenecks are closely tied to the \textit{non-trivial base drift} inherited from the pretrained model. Motivated by this observation, we propose \textbf{Efficient Adjoint Matching (EAM)}, which substantially improves training efficiency by reformulating the SOC problem with a \textit{linear base drift} and a correspondingly modified \textit{terminal cost}. This reformulation removes both sources of inefficiency; it enables training-time sampling with a few-step deterministic ODE solver and yields a closed-form adjoint solution that eliminates backward adjoint simulation. On standard text-to-image reward fine-tuning benchmarks, EAM converges up to 4x faster than AM and matches or surpasses it across various metrics including PickScore, ImageReward, HPSv2.1, CLIPScore and Aesthetics.

扩散模型微调高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。