arXiv:2609.02108cs.CLcs.AI2026-09

提出无需预设长度的高效扩散语言模型补全方法,显著提速且效果更优。

Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models

论文配图:Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models
图 1 · 摘自论文原文
  • 基于探测机制实现无初始长度依赖的补全,避免反复试错
  • 在8个基准上平均提升4.8%通过率(代码)和6.0 BLEU-2(文本)
  • 推理速度比基线快1.82倍,适合对效率敏感的应用场景

扩散语言模型(DLMs)作为自回归范式的有前途替代方案,凭借双向注意力和任意顺序生成能力,天然适用于需要根据前后文生成中间片段的补全任务。然而,补全对片段长度敏感,而传统DLM要求生成前固定长度。尽管已有研究尝试支持动态长度,但仍存在两大局限:(i) 对初始长度敏感,需预设长度启动搜索,易导致次优结果;(ii) 推理效率低,或在生成中插入长度调整操作,或多次通过多步去噪置信度搜索合适长度,带来大量额外前向传播与计算开销。为此,我们提出PILL(Probing-based InfiLling with preset-Length-free decoding),一种无需预设初始长度、额外前向传播极少的高效补全方法,显著降低推理时间。实验表明,在涵盖不同家族、架构和训练方式的五种DLMs及八个补全基准上,PILL相比最强基线平均提升4.8%通过率(代码)和6.0 BLEU-2(文本),同时运行速度快1.82倍。代码已开源:https://github.com/Hsu1023/PILL。

原文摘要 · Abstract (English)

Diffusion language models (DLMs) have emerged as a promising alternative to the auto-regressive paradigm. With bidirectional attention and any-order generation, DLMs naturally fit infilling tasks, which require generating a middle span conditioned on both the prefix and the suffix. However, infilling is sensitive to the length of the span, while DLMs require the length to be fixed before generation. Although prior studies extend DLMs to dynamic lengths, they still suffer from two limitations. (i) Sensitivity to initial length. These methods require a preset length to initialize the search and are highly sensitive to this initial length, often yielding suboptimal results. (ii) Inference inefficiency. They either insert length-changing operations during generation or repeatedly search for an appropriate length using multi-step denoising confidence, both of which introduce substantial extra forward passes and computational cost. Therefore, we propose PILL (Probing-based InfiLling with preset-Length-free decoding), an efficient infilling method for DLMs that requires no preset initial length and adds far fewer extra forward passes than baselines, substantially reducing inference time. Experiments show that, across five DLMs spanning different families, architectures, and training recipes on eight infilling benchmarks, PILL improves over the strongest baseline by +4.8 average pass rate on code and +6.0 BLEU-2 on text, while running 1.82x faster than that baseline. The code is available at https://github.com/Hsu1023/PILL.

扩散模型补全任务高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。