提出自适应依赖结构的扩散语言模型采样策略,提升生成效率与质量。
Adaptation to Intrinsic Dependence in Diffusion Language Models
- 基于数据内在依赖结构动态调整每轮解码的词元数量,无需超参数调优。
- 理论证明收敛速度可达 $ ilde{O}( ext{TC}/K)$ 和 $ ilde{O}( ext{DTC}/K)$,优于以往方法。
- 适合追求高效并行生成、关注推理机制设计的研究者与工程师。
扩散语言模型(DLMs)作为自回归方法的有前景替代方案,实现了超越严格左右顺序的并行词元生成。尽管实证表现持续进步,但关于解码过程中未掩码调度(即每轮解码的词元数量与顺序)如何影响生成质量的理论理解仍有限。本文提出一种分布无关的自适应解码调度,能根据目标数据分布的未知依赖结构自动调整每轮揭示的词元数量,且无需先验知识或超参数调优。与以往固定解码规模的确定性方法不同,本方法在每轮随机化解码词元数。我们证明,在两种特定参数设置下,采样收敛性(以KL散度衡量)分别达到 $ ilde{O}( ext{TC}/K)$ 与 $ ilde{O}( ext{DTC}/K)$。其中 $K$ 为迭代次数,$ ext{TC}$ 与 $ ext{DTC}$ 分别为总相关性和对偶总相关性,刻画了数据内在依赖结构。重要的是,该理论保证适用于实际中常见的 $K<L$ 并行采样场景($L$ 为序列长度)。相比先前理论显著改进,并在低复杂度分布上带来显著采样加速。结果揭示了 DLM 对数据内在结构的自适应能力,也为推理调度设计中引入随机解码规模提供了理论支持。
原文摘要 · Abstract (English)
Diffusion language models (DLMs) have recently emerged as a promising alternative to autoregressive (AR) approaches, enabling parallel token generation beyond a rigid left-to-right order. Despite growing empirical success, the theoretical understanding of how unmasking schedules -- which specify the order and size of unmasked tokens during sampling -- affect generation quality remains limited. In this work, we introduce a distribution-agnostic unmasking schedule for DLMs that adapts to the (unknown) dependence structure of the target data distribution, without requiring any prior knowledge or hyperparameter tuning. In contrast to prior deterministic procedures that fix unmasking sizes, our method randomizes the number of tokens revealed at each iteration. We show that, for two specific parameter choices, the sampling convergence guarantees -- measured by Kullback-Leibler (KL) divergence -- scale as $\widetilde O(\mathsf{TC}/K)$ and $\widetilde O(\mathsf{DTC}/K)$ respectively. Here, $K$ is the number of iterations, and $\mathsf{TC}$ and $\mathsf{DTC}$ are the total correlation and dual total correlation of the target distribution, capturing the intrinsic dependence structure underlying the data. Importantly, our guarantees hold in the practically relevant parallel-sampling regime $K<L$ where $L$ is the token sequence length. These results significantly improve upon prior convergence theories and yield substantial sampling acceleration for low-complexity distributions. Overall, our findings unveil the adaptivity of DLMs to intrinsic data structures and shed light on the benefit of randomized unmasking sizes in inference schedule design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。