通过动态阈值提前终止,让扩散语言模型提速近4倍且几乎不丢分。
Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules
- 基于进度感知的置信度阈值,无需训练即可提前退出采样。
- 指令微调模型加速3.8至4.0倍,性能保留99.8%以上。
- 适用于长文本生成,优于已有基于置信度的早期退出方法。
扩散大语言模型(dLLMs)虽具潜力,但受限于缓慢的迭代采样。本文提出SchED,一种无需训练、模型无关的早期退出算法,通过聚合全跨度对数几率差,在满足平滑的进度依赖置信阈值时即停止解码。我们在两个dLLM系列(Dream与LLaDA)的基座与指令微调版本上,于十项基准任务(包括多选题问答、数学、长文本问答/摘要、翻译)中评估。SchED实现显著且稳定的加速:指令微调模型平均提速3.8–4.0倍,性能保留99.8%–100%;基座模型在保守设置下最高达2.34倍提速,性能保留99.1%–100%。使用重度惩罚质量损失的保守速度指标(QPS, γ=4),证明SchED鲁棒性强,显著优于先前基于置信度的早期退出方法,尤其在长文本生成中表现更优。熵分析显示,指令微调可加速预测熵衰减。通过将真实置信度稳定转化为计算节省,SchED大幅提升了dLLM解码效率。
原文摘要 · Abstract (English)
Diffusion large language models (dLLMs) offer a promising alternative to autoregressive models, but their practical utility is severely hampered by slow, iterative sampling. We present SchED, a training-free, model-agnostic early-exit algorithm that aggregates full-span logit margins and halts decoding once a smooth, progress-dependent confidence threshold is met. We evaluated SchED on two dLLM families (Dream and LLaDA), in base and instruction-tuned variants across ten benchmarks spanning downstream tasks including multiple-choice question answering (MCQ), math, long-form QA/summarization, and translation. SchED delivers large, stable accelerations: on instruction-tuned models, it achieves $3.8$-$4.0\times$ speedups while retaining $99.8$-$100\%$ of the baseline score on average. On base models, SchED yields consistent speedup gains with $99.1$-$100\%$ performance retention, with up to $2.34\times$ under more aggressive settings. Using a conservative speed metric that heavily penalizes quality loss (QPS, $γ{=}4$), we show that SchED is robust and clearly outperforms prior confidence-based early-exit methods, which break down on long-form generation. An entropy analysis of the model's token predictions reveals that instruction tuning speeds up the decay of predictive entropy. By turning genuine confidence stabilization into computational savings, SchED makes dLLM decoding substantially more efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。