arXiv:2609.05727cs.LGcs.AI2026-09

提出统一框架,让生成模型微调与采样一步到位。

Newton Matching for Generative Modeling: A Unified Framework for Fine-Tuning and Sampling

论文配图:Newton Matching for Generative Modeling: A Unified Framework for Fine-Tuning and Sampling
图 1 · 摘自论文原文
  • 用牛顿迭代统一优化生成模型的微调和采样过程。
  • 理论证明可实现全局收敛,且全步长时局部二次收敛。
  • 支持模块化设计,能还原多种现有方法并提升稳定性。

我们提出 Newton Matching,一个用于生成建模中微调与采样的统一框架。目标密度为 $ ho/propto\ u e^{\tau r}$,其中 $r$ 为奖励,$ au>0$ 为逆温度,$ u$ 表示预训练模型的终态密度(微调时)或常数 1(采样时)。将范式从孤立损失转向对标准条件匹配的规范模型进行迭代优化。在兼容的光滑实现假设下,规范速度构成与密度流形微分同胚的流形。将 Fisher-Rao 度量与混合连接传输至该流形,证明反 KL Hessian 等于度量,故牛顿方向即负 Fisher-Rao 梯度。在终态密度 $ ho$ 处,每阶段先执行由正则化奖励 $r - \frac{1}{\tau}\log(\rho/\mu)$ 生成的切向步进,再通过保持终态密度的规范重收缩。此规范回缩实现精确有限步长密度表征。对理想迭代,证明当 $0 < \eta \le \tau$ 时严格反 KL 下降;在弱条件下全局收敛;全步长($\eta=\tau$)时局部二次收敛。协方差与梯度形式分别提供前向或反向回归对构造,均得样本级切向更新损失,具有相同总体最小值,无需重要性采样或完整轨迹反向传播。我们提出近似更新,并定义临界点一致性:仅当 $\rho=\pi$ 时切向位移消失。本框架可精确复现、一致近似或目标修改变体若干代表性方法,支持模块化算法设计。工作推进了生成模型强化学习的理论与算法。

原文摘要 · Abstract (English)

We develop Newton Matching, a unified framework for fine-tuning and sampling in generative modeling. The target is $\pi\propto\mu e^{\tau r}$, where $r$ is the reward, $\tau>0$ the inverse temperature, and $\mu$ denotes the pretrained model's terminal density for fine-tuning or the constant $1$ for sampling. We shift the paradigm from isolated losses to iterative optimization over canonical models: population minimizers of standard conditional matching for terminal densities. Under compatible smooth-realization assumptions, canonical velocities form a manifold diffeomorphic to the density manifold. Transporting the Fisher-Rao metric and mixture connection to this manifold, we show that the reverse-KL Hessian equals the metric, so the Newton direction coincides with the negative Fisher-Rao gradient. At terminal density $\rho$, each stage takes a tangential step generated by the regularized reward $r-\frac1\tau\log(\rho/\mu)$, followed by terminal-density-preserving canonicalization. This canonical retraction yields an exact finite-stepsize density characterization. For the ideal iteration, we prove strict reverse-KL descent away from the target for $0 < \eta \le \tau$, global convergence under mild conditions, and local quadratic convergence for full steps ($\eta=\tau$). Covariance and gradient forms, each with forward or reverse regression-pair constructions, yield sample-wise tangential-update losses with the same population minimizer, without importance sampling or full-trajectory backpropagation. We develop approximate updates and define critical-point consistency as vanishing tangential displacement if and only if $\rho=\pi$. We recover representative methods as exact realizations, critical-point-consistent approximations, or objective-altering variants, enabling modular algorithm design. Our work advances the theory and algorithms of reinforcement learning for generative models.

生成模型强化学习优化框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。