arXiv:2606.22976cs.LGcs.AI2026-06

用图上的随机游走构建可控测试环境,验证并改进掩码扩散模型的并行采样方法。

Understanding Parallel Samplers in Masked Diffusion via Random Walks on Graphs

  • 通过图上随机游走构造可控数据集,模拟掩码扩散模型的潜在结构
  • 证明常见最低熵策略并非总是最优,性能依赖于图结构特性
  • 提出分治式采样器,步骤数对数级增长,训练完美时可保证精确性

本文提出使用图上的随机游走作为可验证的实验平台,研究掩码扩散模型(MDMs)中的不同并行采样策略。我们在固定图的随机游走样本上训练一个MDM,图或转移核不显式提供给模型,仅作为序列中的隐含结构,但具有可控性和可量化评估能力。该框架具备类似数独的验证机制:可验证输出是否为有效游走,并从游走中估计马尔可夫核以衡量分布保真度。利用简单图结构,我们理论证明:基于最低熵等常用评分的并行解掩码并非始终优于随机并行采样器;其性能关键取决于底层图结构。我们提出一种新的分治采样器,步数与序列长度的对数成正比,在理想训练条件下可保证精确性。在多种图游走任务上的实验表明,不同并行采样器在不同图上表现各异。初步实验在预训练的OpenWebText MDM上显示,分治式采样器显著提升了语言生成的速度-质量权衡。这些结果表明,图随机游走是诊断和设计掩码扩散模型并行采样器的机制基准。

原文摘要 · Abstract (English)

In this paper, we propose using random walks on graphs as a verifiable sandbox to study different parallel sampling strategies in masked diffusion models (MDMs). We train an MDM on random walk samples from a fixed graph. The graph or the transition kernel is never shown to the model explicitly and plays the role of latent structure in the sequences, albeit one that is controllable and can be used for quantitative evaluation. Thus, this framework enjoys a Sudoku-like validity check: verifying that an output is a valid walk and estimating the Markov kernel from the walks to measure distribution fidelity. Using simple graphs, we theoretically prove that parallel unmasking via widely used scores like lowest entropy is not uniformly better than a random parallel sampler; the performance critically depends on the structure of the underlying graph. We develop a new bisection sampler for random walks, which takes logarithmic steps in the sequence length and is provably exact under perfect training. Experiments on various graph walk tasks show that different parallel samplers are better for different graphs even in practice. Our initial experiments on a pretrained OpenWebText MDM show that the bisection-style samplers improve speed-quality tradeoffs even for language generation. Together, these results position graph random walks as a mechanistic benchmark for diagnosing and designing parallel samplers for masked diffusion models.

掩码扩散并行采样随机游走模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。