arXiv:2602.23968cs.LG2026-02

用变分推断学习离散扩散模型的生成顺序,提升并行生成效率与质量。

Learning Generation Orders for Masked Discrete Diffusion Models via Variational Inference

  • 通过变分推断框架学习最优生成顺序,支持高效并行采样。
  • 在GSM8K数据集上仅4步生成即达33.1%准确率,优于传统方法。
  • 适合关注并行生成效率与生成质量平衡的研究者。

掩码离散扩散模型(MDMs)是一种有前景的生成建模方法,可实现并行标记生成,相比自回归模型更具效率。然而,在并行生成与样本质量之间实现最优平衡仍是开放问题。当前方法主要依赖固定启发式并行采样策略。虽已有部分基于学习的方法,但从变分推断视角的研究仍不充分。本文提出一种用于学习MDMs并行生成顺序的变分推断框架。我们设计了一种近似后验参数化方式,支持训练时的并行性与高效采样。在GSM8K数据集上的初步实验表明,该方法在高度并行生成场景下表现优异:例如,仅需平均4步生成即达到33.1%准确率,显著优于标准对比方法在相同步数下的23.7–29.0%准确率。我们认为进一步实验与分析将为MDMs的并行生成问题带来深刻见解。

原文摘要 · Abstract (English)

Masked discrete diffusion models (MDMs) are a promising new approach to generative modelling, offering the ability for parallel token generation and therefore greater efficiency than autoregressive counterparts. However, achieving an optimal balance between parallel generation and sample quality remains an open problem. Current approaches primarily address this issue through fixed, heuristic parallel sampling methods. There exist some recent learning based approaches to this problem, but its formulation from the perspective of variational inference remains underexplored. In this work, we propose a variational inference framework for learning parallel generation orders for MDMs. As part of our method, we propose a parameterisation for the approximate posterior of generation orders which facilitates parallelism and efficient sampling during training. Using this method, we conduct preliminary experiments on the GSM8K dataset, where our method performs competitively against heuristic sampling strategies in the regime of highly parallel generation. For example, our method achieves 33.1\% accuracy with an average of only only 4 generation steps, compared to 23.7-29.0\% accuracy achieved by standard competitor methods in the same number of steps. We believe further experiments and analysis of the method will yield valuable insights into the problem of parallel generation with MDMs.

扩散模型并行生成变分推断生成顺序

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。