让扩散语言模型自动调整生成长度,既省算力又提升效果。
Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Models
- 不需训练,通过动态扩展生成长度适应任务需求
- 在多个数据集上达到固定长度模型的性能,且有效令牌率更高
- 适合追求高效生成与灵活长度控制的开发者
扩散大语言模型(DLLMs)作为自回归模型的有力替代,具备并行生成和全局上下文建模能力。然而其实际应用受限于静态预设生成长度:长度不足影响复杂任务表现,过长则带来计算开销甚至性能下降。我们发现模型内部存在与最优输出长度相关的潜在信号。为此提出无需训练的DAEDAL策略,实现扩散模型的动态自适应长度扩展。该方法分两阶段:一是在去噪前从短初始长度出发,基于序列完成度指标迭代扩展至粗粒度适配长度;二是在去噪过程中通过插入掩码标记精准识别并扩展生成不足区域,确保输出完整。大量实验表明,DAEDAL在性能上可媲美甚至超越精心调参的固定长度基线,同时提升有效令牌比例,显著提高计算效率。该方法突破了静态长度限制,使DLLMs更接近自回归模型的能力边界,为高效、智能的生成开辟新路径。
原文摘要 · Abstract (English)
Diffusion Large Language Models (DLLMs) are emerging as a powerful alternative to the dominant Autoregressive Large Language Models, offering efficient parallel generation and capable global context modeling. However, the practical application of DLLMs is hindered by a critical architectural constraint: the need for a statically predefined generation length. This static length allocation leads to a problematic trade-off: insufficient lengths cripple performance on complex tasks, while excessive lengths incur significant computational overhead and sometimes result in performance degradation. While the inference framework is rigid, we observe that the model itself possesses internal signals that correlate with the optimal response length for a given task. To bridge this gap, we leverage these latent signals and introduce DAEDAL, a novel training-free denoising strategy that enables Dynamic Adaptive Length Expansion for Diffusion Large Language Models. DAEDAL operates in two phases: 1) Before the denoising process, DAEDAL starts from a short initial length and iteratively expands it to a coarse task-appropriate length, guided by a sequence completion metric. 2) During the denoising process, DAEDAL dynamically intervenes by pinpointing and expanding insufficient generation regions through mask token insertion, ensuring the final output is fully developed. Extensive experiments on DLLMs demonstrate that DAEDAL achieves performance comparable, and in some cases superior, to meticulously tuned fixed-length baselines, while simultaneously enhancing computational efficiency by achieving a higher effective token ratio. By resolving the static length constraint, DAEDAL unlocks new potential for DLLMs, bridging a critical gap with their Autoregressive counterparts and paving the way for more efficient and capable generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。