arXiv:2607.07026cs.LG2026-07

让扩散语言模型生成符合规则的文本,准确率大幅提升且几乎不增加计算量。

Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata

  • 将约束条件建模为有限自动机,实现对扩散模型的精确高效采样。
  • 在函数调用、数学推理等任务上,准确率提升显著,最高达69%。
  • 适用于并行和分块推理,适合实际部署,延迟增加不足5%。

约束解码对大语言模型服务至关重要,可确保生成内容符合特定格式(如JSON结构的函数调用)。现有方法针对自回归模型设计,假设从左到右逐词生成,并在每一步屏蔽非法词元。而扩散语言模型打破了这一假设:在每个去噪步骤中,同时从全因子化的均值场分布中采样多个位置。本文提出一种精确且高效的算法,可在任意可表示为有限自动机的约束下,对受限均值场后验进行采样。通过将有限自动机视为图模型,我们获得了受约束分布的可计算表示,支持高效推理。该方法在构造上保证约束满足,兼容贪婪与采样解码,且适用于任意重掩码调度下的并行与分块解码。结合算术电路理论中的深度缩减技术,采样深度从线性降低至对数级别。在Dream-7B和LLaDA-8B上的实验表明,在函数调用(xLAM, BFCL)、规划(Sudoku, Countdown)、文本转SQL(Spider)和数学推理(GSM-Symbolic)等多个任务中均有显著准确率提升。例如,在BFCL-Live上,梦境模型(Dream-7B)贪婪解码准确率从63.9%提升至71.5%,随机采样准确率从22.3%升至69.0%——而无约束基线会崩溃,整体墙钟开销低于5%。

原文摘要 · Abstract (English)

Constrained decoding is essential for serving LLMs, ensuring that generated outputs follow specific structures such as JSON schema-formatted function calls. Existing systems are designed for autoregressive models and assume left-to-right generation, masking out invalid next tokens at each step. Diffusion language models, however, break this assumption: they sample multiple positions simultaneously from a fully-factorized mean-field distribution at each denoising step. In this paper, we present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton. Viewing finite automata as graphical models, we obtain tractable representations of the constrained distribution that enable efficient inference. The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding under arbitrary remasking schedules. Applying depth-reduction techniques from arithmetic circuit theory, we further reduce sampling depth from linear to logarithmic in the sequence length. Empirical evaluations on Dream-7B and LLaDA-8B show substantial accuracy gains across various tasks including function calling (xLAM, BFCL), planning (Sudoku, Countdown), text-to-SQL (Spider), and math reasoning (GSM-Symbolic), with little inference overhead relative to unconstrained decoding. For example, on BFCL-Live, our approach improves Dream-7B's greedy decoding accuracy from 63.9% to 71.5%, and stochastic sampling accuracy from 22.3% to 69.0%, where the unconstrained baseline collapses, with under 5% wall-clock overhead.

扩散模型约束解码自动机生成质量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。