让扩散语言模型生成符合规则的结构化文本，提升可靠性。
DINGO: Constrained Inference for Diffusion LLMs
- 用动态规划实现约束解码，保持输出分布真实
- 在数学和JSON生成任务中准确率提升68个百分点
- 适合需要严格格式输出的应用场景
扩散语言模型（Diffusion LLMs）作为传统自回归模型的有前景替代方案，具备显著提升推理效率的潜力。然而，现有扩散模型无法证明性地施加用户指定的形式约束（如正则表达式），使其在需要结构化输出的任务（如固定模式JSON生成）中不可靠。与逐个生成标记的自回归模型不同，扩散模型并行预测一组标记，这使得为顺序生成设计的传统约束解码算法无法保留真实的输出分布。为此，我们提出DINGO——一种基于动态规划的约束解码策略，兼具高效性与分布保真性。DINGO能够在模型预测分布下采样概率最高的满足任意用户指定正则表达式的输出字符串。在标准符号数学和JSON生成基准测试中，DINGO相较于无约束推理最高提升68个百分点。
原文摘要 · Abstract (English)
Diffusion LLMs have emerged as a promising alternative to conventional autoregressive LLMs, offering significant potential for improved runtime efficiency. However, existing diffusion models lack the ability to provably enforce user-specified formal constraints, such as regular expressions, which makes them unreliable for tasks that require structured outputs, such as fixed-schema JSON generation. Unlike autoregressive models that generate tokens sequentially, diffusion LLMs predict a block of tokens in parallel. This parallelism makes traditional constrained decoding algorithms, which are designed for sequential token prediction, ineffective at preserving the true output distribution. To address this limitation, we propose DINGO, a dynamic programming-based constrained decoding strategy that is both efficient and provably distribution-preserving. DINGO enables sampling of output strings with the highest probability under the model's predicted distribution, while strictly satisfying any user-specified regular expression. On standard symbolic math and JSON generation benchmarks, DINGO achieves up to a 68 percentage point improvement over unconstrained inference
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。