arXiv:2601.22527cs.CL2026-01被引 8

无需训练即可动态调整生成长度,提升掩码扩散语言模型效率

$ρ$-$\texttt{EOS}$: Training-free Bidirectional Variable-Length Control for Masked Diffusion LLMs

  • 通过隐式EOS密度实时判断生成是否充分,指导长度双向调节
  • 在数学与代码任务上保持性能的同时,推理效率显著提升
  • 无需额外训练或迭代,适合追求高效推理的部署场景

当前掩码扩散大语言模型(masked dLLMs,如 LLaDA)虽支持并行生成和全局建模,但存在生成长度固定的根本缺陷,导致输出质量与计算效率难以兼顾。本文研究去噪过程发现,结束符(EOS)token的隐式密度(ρ)可作为生成充分性的可靠信号。该密度随去噪过程演变,能判断当前掩码空间是否过剩或不足,从而指引长度调整方向。基于此,提出ρ-EOSE,一种无需训练、单阶段的双向可变长度生成策略。相比以往两阶段方法(需分步调整长度与插入掩码),ρ-EOSE在统一去噪过程中持续估算隐式EOS密度:密度过高触发MASK收缩,过低则诱导扩展。在数学与代码基准上的大量实验表明,ρ-EOSE在保持性能的同时,显著提升推理效率与标记利用率。代码已开源。

原文摘要 · Abstract (English)

Beyond parallel generation and global context modeling, current masked diffusion large language models (masked dLLMs, i.e., LLaDA) suffer from a fundamental limitation: they require a predefined, fixed generation length, which lacks flexibility and forces an inevitable trade-off between output quality and computational efficiency. To address this, we study the denoising dynamics and find that the implicit density ($ρ$) of end-of-sequence ($\texttt{EOS}$) tokens serves as a reliable signal of generation sufficiency. In particular, the evolving implicit $\texttt{EOS}$ density during denoising reveals whether the current masked space is excessive or insufficient, thereby guiding the adjustment direction for generation length. Building on this insight, we propose $\textbf{$ρ$-$\texttt{EOS}$}$, a training-free, single-stage strategy that enables bidirectional variable-length generation for masked dLLMs. Unlike prior two-stage approaches--which require separate length adjustment and iterative mask insertion phases while supporting only unidirectional expansion--$\textbf{$ρ$-$\texttt{EOS}$}$ achieves bidirectional length adjustment within a unified denoising process by continuously estimating the implicit $\texttt{EOS}$ density: excessively high density triggers $\texttt{MASK}$ token contraction, while insufficient density induces expansion. Extensive experiments on mathematics and code benchmarks demonstrate that $\textbf{$ρ$-$\texttt{EOS}$}$ achieves comparable performance while substantially improving inference efficiency and token utilization. Code is available at https://github.com/yjyddq/rho-EOS.

扩散模型生成控制推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。