通过消除时空冗余,让扩散语言模型生成速度提升近90%。
$R^2$-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction

- 设计无需训练的解码规则,合并局部置信度与预测结果。
- 减少高达88%的解码步骤,生成质量保持不变。
- 适合追求高效推理的模型部署者与研究者。
扩散语言模型(dLLMs)作为并行生成的替代方案,虽能实现同时预测多个词元,但实际解码仍存在高延迟问题。本文发现,该问题主要源于解码过程中的时空冗余:空间上因置信度聚类和位置模糊导致重复计算,时间上因反复重掩码已稳定的预测项。为此提出R²-dLLM框架,从训练与推理双角度减少冗余。推理时引入免训练的解码规则,聚合局部置信度与词元预测,并提前确定稳定词元以避免重复步骤;进一步提出一种冗余感知的监督微调流程,使模型适应高效解码路径,降低对人工阈值的依赖。实验表明,R²-dLLM相较现有策略最多减少88%解码步骤,且在不同模型与任务中保持良好生成质量。结果验证了解码冗余是dLLMs的核心瓶颈,显式消除冗余可带来显著实用效率提升。代码与模型已在https://github.com/GATECH-EIC/R2-dLLM发布。
原文摘要 · Abstract (English)
Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to autoregressive generation by enabling parallel token prediction. However, practical dLLM decoding still suffers from high inference latency, which limits deployment. In this work, we observe that a substantial part of this inefficiency comes from recurring redundancy in the decoding process, including spatial redundancy caused by confidence clusters and positional ambiguity, and temporal redundancy caused by repeatedly remasking predictions that have already stabilized. Motivated by these patterns, we propose $R^{2}$-dLLM, a unified framework for reducing decoding redundancy from both inference and training perspectives. At inference time, we introduce training-free decoding rules that aggregate local confidence and token predictions, and finalize temporally stable tokens to avoid redundant decoding steps. We further propose a redundancy-aware supervised fine-tuning pipeline that aligns the model with efficient decoding trajectories and reduces reliance on manually tuned thresholds. Experiments demonstrate that $R^{2}$-dLLM consistently reduces the number of decoding steps by up to 88\% compared to existing decoding strategies, while maintaining competitive generation quality across different models and tasks. These results validate that decoding redundancy is a central bottleneck in dLLMs, and that explicitly reducing it yields substantial practical efficiency gains. Our code and models are available at https://github.com/GATECH-EIC/R2-dLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。