arXiv:2604.06330cs.CL2026-04被引 1

通过时空稳定性提升扩散语言模型的生成速度。

STDec: Spatio-Temporal Stability Guided Decoding for dLLMs

论文配图:STDec: Spatio-Temporal Stability Guided Decoding for dLLMs
图 1 · 摘自论文原文
  • 基于邻近词和多步预测一致性动态调整解码阈值。
  • 在MBPP上实现14.17倍加速,任务得分基本不变。
  • 无需训练,兼容缓存加速,适合高效推理场景。

扩散大语言模型(dLLMs)发展迅速,被视为自回归范式的有前景替代方案。然而,多数dLLM解码器仍采用全局置信度阈值,未显式建模邻近解码状态的局部上下文或跨步骤的预测词元ID时间一致性。为此,我们提出一种简单的时空稳定性引导解码方法STDec。观察发现,dLLM解码中存在强时空稳定性:新解码词元通常位于已解码邻居附近,且其预测词元ID在多个去噪步骤中保持一致。受此启发,STDec引入空间感知解码与时间感知解码。空间感知解码通过聚合邻近词的解码状态动态生成词元自适应阈值;时间感知解码对连续多步预测相同的词元放宽解码阈值。该方法无需训练,且与基于缓存的加速方法兼容。在文本推理与多模态理解基准测试中,STDec显著提升吞吐量,同时保持相近的任务性能。特别地,在使用LLaDA的MBPP上,速度最高提升14.17倍,性能相当。

原文摘要 · Abstract (English)

Diffusion Large Language Models (dLLMs) have achieved rapid progress, viewed as a promising alternative to the autoregressive paradigm. However, most dLLM decoders still adopt a global confidence threshold, and do not explicitly model local context from neighboring decoded states or temporal consistency of predicted token IDs across steps. To address this issue, we propose a simple spatio-temporal stability guided decoding approach, named STDec. We observe strong spatio-temporal stability in dLLM decoding: newly decoded tokens tend to lie near decoded neighbors, and their predicted IDs often remain consistent across several denoising steps. Inspired by this stability, our STDec includes spatial-aware decoding and temporal-aware decoding. The spatial-aware decoding dynamically generates the token-adaptive threshold by aggregating the decoded states of nearby tokens. The temporal-aware decoding relaxes the decoding thresholds for tokens whose predicted token IDs remain consistent over denoising steps. Our STDec is training-free and remains compatible with cache-based acceleration methods. Across textual reasoning and multimodal understanding benchmarks, STDec substantially improves throughput while maintaining comparable task performance score. Notably, on MBPP with LLaDA, STDec achieves up to 14.17x speedup with a comparable score. Homepage: https://yzchen02.github.io/STDec.

扩散模型解码加速推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。