arXiv:2510.18165cs.AIcs.CL2025-10中稿 · ACL被引 16

提升扩散语言模型代码生成速度与质量,动态调整解码策略。

Saber: An Efficient Sampling with Adaptive Acceleration and Backtracking Enhanced Remasking for Diffusion Language Model

  • 根据模型置信度动态调整每步解码的掩码数量,实现自适应加速。
  • 引入回溯机制修复因上下文变化导致的错误预测,准确率平均提升1.9%。
  • 无需训练即可显著提速251.4%,适合追求高效高质代码生成的场景。

扩散语言模型(DLMs)作为自回归模型的有力替代,具备并行生成和双向上下文建模的优势。然而,在代码生成等结构化任务中,其推理速度与输出质量间存在严重权衡:减少采样步骤虽能加速,但常导致性能崩溃。我们发现根本原因在于:1)结构化序列解码中生成难度分布不均,静态加速策略效果不佳;2)生成过程中上下文持续演化,早期高置信预测易变为不可逆错误。本文提出无需训练的采样算法Saber,首次在代码生成中同时实现更优的速度与质量。Saber基于模型置信度动态调整每步解码的掩码数,并通过回溯机制回退因新上下文导致置信度下降的词元。理论分析支持其有效性。在多个主流代码生成基准上,相比主流DLM采样方法,Saber平均提升Pass@1准确率1.9%,推理速度平均提升251.4%。该工作显著缩小了扩散模型与自回归模型在代码生成上的性能差距。

原文摘要 · Abstract (English)

Diffusion language models (DLMs) are emerging as a compelling alternative to the dominant autoregressive paradigm, offering inherent advantages in parallel generation and bidirectional context modeling. However, for the tasks with strict structural constraints such as code generation, DLMs face a critical trade-off between inference speed and output quality, where accelerating generation by reducing sampling steps often leads to catastrophic performance collapse. We find that the fundamental reasons are: 1) the generation difficulty is non-uniform in the structured sequence decoding steps, making DLM's static acceleration strategy suboptimal; 2) the context of tokens generated by DLM evolves continuously, causing early high-confidence predictions to turn into irreversible errors. In this paper, we introduce efficient Sampling with Adaptive acceleration and Backtracking Enhanced Remasking (i.e., Saber), a novel training-free sampling algorithm for DLMs that first achieves both better inference speed and output quality in code generation. Saber dynamically adjusts the number of tokens unmasked per step based on the model's evolving confidence, and utilizes a backtracking mechanism to revert tokens whose confidence drops as new context emerges, with its effectiveness supported by theoretical analysis. Extensive experiments on multiple mainstream code generation benchmarks show that Saber boosts Pass@1 accuracy by an average of 1.9\% over mainstream DLM sampling methods, while achieving an average 251.4\% inference speedup. By leveraging the inherent advantages of DLMs, our work significantly narrows the performance gap with autoregressive models in code generation.

扩散模型代码生成采样优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。