让扩散语言模型并行生成更连贯，提升代码生成速度与质量
Locally Coherent Parallel Decoding in Diffusion Language Models
- 用小型自回归模型处理局部依赖，实现并行生成中的语法连贯
- 0.6B参数小模型即可消除生成错误，代码生成准确率更高
- 适合需要快速生成高质量代码的开发者和编辑场景
扩散语言模型（DLMs）作为自回归模型的替代方案，具备亚线性生成延迟和双向建模能力，特别适用于代码生成与编辑。实现离散DLM的亚线性延迟需并行预测多个词元，但标准DLM独立采样各词元，忽略并发生成词元间的联合依赖，导致语法不一致和多词元结构破坏。本文提出CoDiLA（局部自回归协同扩散），将局部解码交由一个小型辅助自回归模型在扩散潜空间中处理，既保持并行生成效率，又确保块内序列有效性，同时保留跨块双向建模能力。实验表明，仅使用0.6B参数的辅助模型即可有效消除连贯性缺陷，在代码生成基准上建立新的精度-速度权衡前沿。
原文摘要 · Abstract (English)
Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) models, offering sub-linear generation latency and bidirectional capabilities that are particularly appealing for code generation and editing. Achieving sub-linear latency in discrete DLMs requires predicting multiple tokens in parallel. However, standard DLMs sample tokens independently from conditional marginal distributions, failing to capture the joint dependencies among concurrently generated tokens. As a result, they often lead to syntactic inconsistencies and break multi-token structures. In this work, we introduce CoDiLA (Coherent Diffusion with Local Autoregression), a method that reconciles parallel sampling with local dependency modeling. Rather than forcing the DLM to resolve fine-grained syntax, CoDiLA delegates local decoding to a small, auxiliary AR model operating on the diffusion latents. This design allows for parallel generation while ensuring sequential validity within a block and maintaining core DLM capabilities, including bidirectional modeling across blocks. We demonstrate that using a highly compact auxiliary AR model (e.g., 0.6B parameters) effectively eliminates coherence artifacts, establishing a new Pareto frontier for accuracy and speed in code generation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。