提出动态采样策略，让扩散语言模型生成速度提升34倍。
Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principles
- 通过自适应切换探索与加速阶段，动态调整采样策略。
- 在LLaDA上实现15.63倍加速，结合缓存达34.22倍。
- 适合追求高速生成且对延迟敏感的部署场景。
基于扩散的语言模型（dLLMs）通过并行生成令牌显著降低推理延迟，成为传统自回归大模型的有力替代。然而，现有采样策略如基于置信度或半自回归解码存在行为静态问题，导致效率不足且灵活性差。本文提出SlowFast Sampling，一种新型动态采样策略，通过自适应交替探索与加速解码阶段实现高效生成。该方法遵循三个黄金原则：确定性原则、收敛性原则和位置原则，指导何时何地可安全高效解码。进一步与dLLM-Cache结合，减少冗余计算。大量实验表明，该方法在不同基准和模型上实现最高15.63×加速，结合缓存可达34.22×。尤其值得注意的是，其吞吐量超越强自回归基线（如LLaMA3 8B），证明精心设计的采样策略可充分释放dLLMs在快速高质量生成中的潜力。
原文摘要 · Abstract (English)
Diffusion-based language models (dLLMs) have emerged as a promising alternative to traditional autoregressive LLMs by enabling parallel token generation and significantly reducing inference latency. However, existing sampling strategies for dLLMs, such as confidence-based or semi-autoregressive decoding, often suffer from static behavior, leading to suboptimal efficiency and limited flexibility. In this paper, we propose SlowFast Sampling, a novel dynamic sampling strategy that adaptively alternates between exploratory and accelerated decoding stages. Our method is guided by three golden principles: certainty principle, convergence principle, and positional principle, which govern when and where tokens can be confidently and efficiently decoded. We further integrate our strategy with dLLM-Cache to reduce redundant computation. Extensive experiments across benchmarks and models show that SlowFast Sampling achieves up to 15.63$\times$ speedup on LLaDA with minimal accuracy drop, and up to 34.22$\times$ when combined with caching. Notably, our approach outperforms strong autoregressive baselines like LLaMA3 8B in throughput, demonstrating that well-designed sampling can unlock the full potential of dLLMs for fast and high-quality generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。