arXiv:2602.15008cs.LGcs.IT2026-02中稿 · the Conference on …被引 17

提出离散扩散模型采样新算法,理论证明其收敛速度与数据结构自适应。

Efficient Sampling with Discrete Diffusion Models: Sharp and Adaptive Guarantees

  • 基于连续时间马尔可夫链建模,改进τ-跳跃采样算法。
  • 对均匀和掩码噪声过程均实现ε精度的KL散度收敛,复杂度达$ ilde O(d/)$。
  • 无需先验知识即可自动适应低维结构,适合高维稀疏数据场景。

离散空间上的扩散模型近期表现出色,但理论基础仍不完善。本文在连续时间马尔可夫链框架下研究基于分数的离散扩散模型采样效率,重点分析τ-跳跃采样器。针对均匀和掩码加噪过程,建立了达到ε精度的KL散度收敛的严格保证。对于均匀离散扩散,τ-跳跃算法的迭代复杂度为$ ilde O(d/)$,消除了与词表大小$S$的线性依赖,相比已有结果提升$O(d)$倍;同时证明了该复杂度下界,表明在一般情况下对环境维度的线性依赖不可避免。对于掩码离散扩散,引入改进的τ-跳跃采样器,其收敛速率由一个信息论量——有效总相关性决定,该量上限为$d \ log S$,但在结构化数据(如隐马尔可夫模型、图像数据、随机图)中可为亚线性甚至常数。因此,采样器可无须事先知晓结构即自适应,实现亚线性收敛率。分析不依赖于分数估计器的有界性或光滑性假设,仅需控制分数熵损失。

原文摘要 · Abstract (English)

Diffusion models over discrete spaces have recently shown striking empirical success, yet their theoretical foundations remain incomplete. In this paper, we study the sampling efficiency of score-based discrete diffusion models under a continuous-time Markov chain (CTMC) formulation, with a focus on $τ$-leaping-based samplers. We establish sharp convergence guarantees for attaining $\varepsilon$ accuracy in Kullback-Leibler (KL) divergence for both uniform and masking noising processes. For uniform discrete diffusion, we show that the $τ$-leaping algorithm achieves an iteration complexity of order $\tilde O(d/\varepsilon)$, with $d$ the ambient dimension of the target distribution, eliminating linear dependence on the vocabulary size $S$ and improving existing bounds by a factor of $d$; moreover, we establish a matching algorithmic lower bound showing that linear dependence on the ambient dimension is unavoidable in general. For masking discrete diffusion, we introduce a modified $τ$-leaping sampler whose convergence rate is governed by an intrinsic information-theoretic quantity, termed the effective total correlation, which is bounded by $d \log S$ but can be sublinear or even constant for structured data. As a consequence, the sampler provably adapts to low-dimensional structure without prior knowledge or algorithmic modification, yielding sublinear convergence rates for various practical examples (such as hidden Markov models, image data, and random graphs). Our analysis requires no boundedness or smoothness assumptions on the score estimator beyond control of the score entropy loss.

扩散模型采样效率理论分析离散空间

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。