arXiv:2602.00612cs.CL2026-02被引 3

提出LAVE方法,让扩散语言模型生成代码时更可靠地保持语法正确。

Lookahead-then-Verify: Reliable Constrained Decoding for Diffusion LLMs under Context-Free Grammars

  • 先预判再验证:利用模型并行预测能力提前检查每步生成的合理性
  • 在4个模型、3个基准上显著提升语法正确率,几乎不增加计算开销
  • 适合需要高可靠性输出的场景,如代码生成、化学式构造

扩散大语言模型(dLLMs)在生成源代码、化学表达式等上下文无关文法定义的形式语言方面展现出良好生成能力。然而作为概率模型,其生成结果仍难以保证语法正确。一种有前景的解决思路是引入约束解码以强制语法正确性。但现有方法面临两大挑战:一是dLLMs的非自回归特性使多数传统约束解码方法不适用;二是现有专为dLLMs设计的方法可能允许无法完成的有效句子的中间输出,严重影响实用性。为此,本文提出LAVE,一种专为dLLMs设计的约束解码方法。该方法利用dLLMs在每次前向传播中可并行预测所有位置词元分布的特性,在模型提出新词元时,通过这些分布进行前瞻验证,高效可靠地判断该词元是否可被后续扩展为合法句式。此设计确保了中间状态始终具备完成为有效句子的潜力,从而实现可靠约束。在四个主流dLLMs和三个代表性基准上的大量实验表明,LAVE持续优于现有基线,显著提升语法正确性,且运行开销可忽略不计。

原文摘要 · Abstract (English)

Diffusion Large Language Models (dLLMs) have demonstrated promising generative capabilities and are increasingly used to produce formal languages defined by context-free grammars, such as source code and chemical expressions. However, as probabilistic models, they still struggle to generate syntactically valid outputs reliably. A natural and promising direction to address this issue is to adapt constrained decoding techniques to enforce grammatical correctness during generation. However, applying these techniques faces two primary obstacles. On the one hand, the non-autoregressive nature of dLLMs renders most existing constrained decoding approaches inapplicable. On the other hand, current approaches specifically designed for dLLMs may allow intermediate outputs that are impossible to complete into valid sentences, which significantly limits their reliability in practice. To address these challenges, we present LAVE, a constrained decoding approach specifically designed for dLLMs. Our approach leverages a key property of dLLMs, namely their ability to predict token distributions for all positions in parallel during each forward pass. Whenever a new token is proposed by model, LAVE performs lookahead using these distributions to efficiently and reliably verify the validity of the proposed token. This design ensures reliable constraints by reliably preserving the potential for intermediate outputs to be extended into valid sentences. Extensive experiments across four widely used dLLMs and three representative benchmarks demonstrate that LAVE consistently outperforms existing baselines and achieves substantial improvements in syntactic correctness, while incurring negligible runtime overhead.

扩散模型约束解码语法正确代码生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。