通过首步置信度提前确定简单词元，加速扩散语言模型推理。
CAI-DLLM: Convergence Aware Inference for Diffusion Language Models
- 基于首步置信度动态分配去噪步骤，优先处理困难词元。
- 在LLaDA上实现18.2倍加速，数学任务准确率提升至77.41%。
- 无需重训练，适合高效率生成场景，如长文本与复杂推理。
扩散语言模型可并行生成多个词元，但推理仍需反复去噪，成本高昂，尤其在重复计算已稳定的词元时。为此，我们提出无需训练的CAI-DLLM推理方法，利用首步置信度引导去噪过程，提前确定简单词元，为困难词元分配更多去噪步骤，并调整输出块间的解码调度。该方法仅依赖首步置信度信号，无需重新训练、额外预测器或权重更新。我们在LLaDA-8B-Instruct和Dream-7B-Instruct上评估了其在数学、代码、推理、常识及长上下文任务上的表现。CAI-DLLM在LLaDA GSM8K上实现最高18.2倍的时钟速度提升，准确率从76.27%提高到77.41%；在Dream HumanEval上达到13.1倍加速，pass@1达48.17%，优于无缓存推理的46.95%。在更难的推理任务中，速度提升达44.8倍，准确率最大下降4.4点，能耗降低最多95.3%。
原文摘要 · Abstract (English)
Diffusion language models can generate many tokens in parallel, but they still require repeated denoising steps during inference. This makes generation costly, especially when the model continues to recompute tokens that are already stable. To address these limitations, we propose CAI-DLLM, a training-free inference method that uses first-step confidence to guide denoising and reduce inference time. Specifically, CAI-DLLM commits easy tokens earlier, allocates more denoising steps to harder tokens, and adjusts decoding schedules across output blocks. As it relies only on first-step confidence signals, it does not require retraining, extra predictors, or weight updates. We evaluate CAI-DLLM on LLaDA-8B-Instruct and Dream-7B-Instruct across math, code, reasoning, commonsense, and long-context tasks. CAI-DLLM achieves up to 18.2x wall clock inference speedup on LLaDA GSM8K while improving accuracy from 76.27% to 77.41%, and up to 13.1x speedup on Dream HumanEval while achieving higher pass@1 than no-cache inference, 48.17% compared with 46.95%. On harder reasoning tasks, speedups reach 44.8x, with a largest accuracy drop of 4.4 points, while energy consumption is reduced by up to 95.3%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。