arXiv:2602.07546cs.CLcs.LG2026-02被引 4

解决扩散语言模型变长生成中的长度偏差问题,提升生成可靠性。

Improving Variable-Length Generation in Diffusion Language Models via Length Regularization

  • 引入显式长度正则化,修正生成置信度的长度偏差。
  • 在未知长度下实现51.3% Pass@1(HumanEvalInfilling),较DreamOn提升13.4%。
  • 无需修改模型或训练,可动态调整生成长度,适合实际应用。

扩散大语言模型(DLLMs)在变长生成任务中存在固有缺陷,因其推理基于固定长度的画布,隐含假设目标长度已知。当长度未知时,如真实场景下的补全与填充任务,直接比较不同掩码长度的置信度会产生系统性偏差,导致生成不足或冗余。本文揭示该失败源于生成置信度中内在的长度偏差,使现有DLLMs无法可靠确定生成长度。为此,我们提出LR-DLLM:一种针对DLLMs的长度正则化推理框架,将生成长度视为显式变量,在不修改底层模型或训练流程的前提下,通过显式长度正则化解耦语义兼容性与长度相关不确定性,从而实现可靠的长度判断。该方法支持动态扩展或收缩生成范围。实验表明,LR-DLLM在完全未知长度的HumanEvalInfilling任务上达到51.3% Pass@1(相比DreamOn提升13.4%),在四语言McEval上的平均Pass@1为51.5%(相比DreamOn提升14.3%)。

原文摘要 · Abstract (English)

Diffusion Large Language Models (DLLMs) are inherently ill-suited for variable-length generation, as their inference is defined on a fixed-length canvas and implicitly assumes a known target length. When the length is unknown, as in realistic completion and infilling, naively comparing confidence across mask lengths becomes systematically biased, leading to under-generation or redundant continuations. In this paper, we show that this failure arises from an intrinsic lengthinduced bias in generation confidence estimates, leaving existing DLLMs without a robust way to determine generation length and making variablelength inference unreliable. To address this issue, we propose LR-DLLM, a length-regularized inference framework for DLLMs that treats generation length as an explicit variable and achieves reliable length determination at inference time. It decouples semantic compatibility from lengthinduced uncertainty through an explicit length regularization that corrects biased confidence estimates. Based on this, LR-DLLM enables dynamic expansion or contraction of the generation span without modifying the underlying DLLM or its training procedure. Experiments show that LRDLLM achieves 51.3% Pass@1 on HumanEvalInfilling under fully unknown lengths (+13.4% vs. DreamOn) and 51.5% average Pass@1 on four-language McEval (+14.3% vs. DreamOn).

扩散模型变长生成长度正则化代码生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。