arXiv:2605.11361cs.LGcs.DS2026-05被引 1

探索扩散模型对齐的可计算性边界,揭示不同距离度量下的高效实现方法。

The tractability landscape of diffusion alignment: regularization, rewards, and computational primitives

论文配图:The tractability landscape of diffusion alignment: regularization, rewards, and computational primitives
图 1 · 摘自论文原文
  • 以简单算法原语为出发点,研究奖励对齐的可行实现路径。
  • 在KL距离下,线性指数倾斜可高效采样凸低维奖励。
  • 在Wasserstein距离下,近端传输口可高效处理凹或低维利普希茨奖励。

推理时奖励对齐旨在将预训练扩散模型的基础分布 $p$ 转化为偏好奖励 $r$ 的采样器,同时保持与 $p$ 的接近性。由于缺乏统一的分布距离度量,不同选择导致不同的“奖励对齐”分布及相应的算法问题。本文提出基于原语的方法:不假设任意对齐分布可采样,而是追问哪些简单算法原语足以实现非平凡奖励类的对齐。当使用KL距离时,目标分布为 $q(x) /propto p(x) \ ext{exp}(λ^{-1}r(x))$,我们证明形式为 $q(x)\ ext{propto} p(x)\ ext{exp}(\langle θ, x \rangle)$ 的线性指数倾斜——根据近期工作 [MRR26] 可高效采样——是实现广泛凸低维奖励对齐的充分原语。当使用Wasserstein距离时,对应原语为近端传输口:给定 $x$,求解 $\mbox{argmax}_y \{r(y)- λc(x,y)\}$。该口可对凹或低维利普希茨奖励 $r(x)=f(Ax)$ 高效实现。结果表明,对齐所用分布距离直接影响所需计算原语与可处理奖励类。

原文摘要 · Abstract (English)

Inference-time reward alignment asks how to turn a pre-trained diffusion model with base law $p$ into a sampler that favors a reward $r$ while remaining close to $p$. Since there is no canonical distributional distance for this closeness constraint, different choices lead to different "reward-aligned" laws and, just as importantly, different algorithmic problems. We develop a primitive-based approach to reward alignment: rather than assuming arbitrary reward-aligned laws can be sampled, we ask which simple algorithmic primitives suffice to implement alignment for non-trivial reward classes. If closeness is measured in KL distance, the target law is $q(x) \propto p(x) \exp(λ^{-1}r(x))$. For this setting, we show that linear exponential tilts of the form $q(x)\propto p(x)\exp(\langle θ, x \rangle)$ -- which according to recent work [MRR26] can be efficiently sampled from -- are a sufficient primitive for aligning to a very broad class of convex low-dimensional rewards. If closeness is measured in Wasserstein distance, the corresponding primitive is a proximal transport oracle: given $x$, solve $\mbox{argmax}_y \{r(y)- λc(x,y)\}$. This oracle can be efficiently implemented for concave or low-dimensional Lipschitz rewards $r(x)=f(Ax)$. Together, these results illustrate that the choice of distribution distance for alignment affects the computational primitive and the tractable reward class.

扩散模型奖励对齐可计算性优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。