提出轨迹级推测解码，提升扩散语言模型生成速度。
Trajectory-Level Speculative Decoding for Diffusion Language Models
- 基于置信度分层树搜索构造候选去噪轨迹
- 减少30-40%去噪迭代，单步生成量从2.6增至4.3
- 适合追求高效推理的生成任务研究者
基于扩散的语言模型（dLLMs）通过迭代去噪实现并行生成，但现有解码策略在置信度低时退化为单标记生成，严重限制吞吐量。与自回归模型中按固定左右顺序进行推测解码不同，dLLMs需对带有显式位置和解掩码顺序的多标记更新序列——去噪轨迹——进行推测。本文提出一种轨迹级推测框架，通过置信度分层树探索构建草稿去噪轨迹，并利用双向注意力掩码进行块内并行验证。该方法进一步引入块间推测，利用扩散模型的双向结构实现跨块前瞻。我们形式化刻画了该方法精确成立的条件，并指出轨迹漂移是并行性提升的根本代价。基于Fast-dLLM的双缓存架构，本框架使去噪迭代减少30-40%，每步生成令牌数从2.6提升至4.3，相较原生dLLMs提速7-14倍，比Fast-dLLM快1.3倍，且在推理与代码基准上准确率变化低于1%。
原文摘要 · Abstract (English)
Diffusion-based language models (dLLMs) enable parallel token generation through iterative denoising, but existing decoding strategies collapse to single-token generation under low confidence, severely limiting throughput. Unlike autoregressive models where speculative decoding operates on token sequences in a fixed left-to-right order, dLLMs require speculating over denoising trajectories-sequences of multi-token updates with explicit positions and unmasking orders. We develop a trajectory-level speculative framework that constructs draft denoising trajectories via confidence-stratified tree exploration and verifies them through blockwise parallel evaluation with bidirectional attention masking. Our method further introduces inter-block speculation, exploiting diffusion models' bidirectional structure to perform cross-block lookahead. We formally characterize when this approach is exact and identify trajectory drift as the fundamental cost of increased parallelism. Building on Fast-dLLM's dual-cache infrastructure, our framework reduces denoising iterations by 30-40% and increases tokens-per-step from 2.6 to 4.3, achieving 7-14x speedup over vanilla dLLMs and 1.3x over Fast-dLLM with less than 1% accuracy change across reasoning and code benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。