首次实现扩散语言模型的语法约束生成,确保代码和数据格式正确。
Constrained Decoding of Diffusion LLMs with Context-Free Grammars
- 将语法约束解码转为加法填充问题,支持多区域填充。
- 在C++和JSON生成中达到近乎100%语法正确率。
- 适用于需严格格式输出的场景,如代码补全与结构化数据提取。
大型语言模型在多个领域表现优异,但其概率性生成难以保证代码补全、结构化数据提取等任务中的语法正确性。现有约束解码方法不适用于新兴的扩散语言模型(diffusion LLMs)。本文提出首个针对上下文无关文法(CFG)的扩散模型约束解码方法。通过将约束解码转化为更通用的加法填充问题,并将其归约为判断目标语言与正则语言交集是否为空,设计出高效算法求解上下文无关语言。在C++代码补全和JSON结构化数据提取等任务上的实验表明,该方法在保持或提升功能正确性的前提下,实现了接近100%的语法正确率。重要的是,效率优化使计算开销保持在实际可用范围内。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown promising performance across diverse domains. Many practical applications of LLMs, such as code completion and structured data extraction, require adherence to syntactic constraints specified by a formal language. Yet, due to their probabilistic nature, LLM output is not guaranteed to adhere to such formal languages. Prior work has proposed constrained decoding as a means to restrict LLM generation to particular formal languages. However, existing works are not applicable to the emerging paradigm of diffusion LLMs, when used in practical scenarios such as the generation of formally correct C++ or JSON output. In this paper we address this challenge and present the first constrained decoding method for diffusion models, one that can handle formal languages captured by context-free grammars. We begin by reducing constrained decoding to the more general additive infilling problem, which asks whether a partial output can be completed to a valid word in the target language. This problem also naturally subsumes the previously unaddressed multi-region infilling constrained decoding. We then reduce this problem to the task of deciding whether the intersection of the target language and a regular language is empty and present an efficient algorithm to solve it for context-free languages. Empirical results on various applications, such as C++ code infilling and structured data extraction in JSON, demonstrate that our method achieves near-perfect syntactic correctness while consistently preserving or improving functional correctness. Importantly, our efficiency optimizations ensure that the computational overhead remains practical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。