让扩散模型并行生成多个词元,提升推理速度。
Divide and Conquer: Accelerating Diffusion-Based Large Language Models via Adaptive Parallel Decoding
- 分治策略:先分组再并行解码,逐步逼近完整输出
- 实测推理速度提升显著,生成质量与现有方法相当
- 适合追求高效推理的扩散模型应用者
基于扩散的大语言模型(dLLMs)在各类推理任务中表现出色,成为自回归模型的替代方案。与逐个生成词元的自回归模型不同,dLLMs理论上可在每一步并行生成多个词元。然而,当前dLLMs仍多采用单词元生成,因直接并行解码多个掩码词元常导致生成质量下降和不稳定。这反映出理论并行性与实际性能之间的巨大差距。为此,本文提出自适应并行解码方法DiCo,采用三阶段分治范式以释放dLLMs的固有并行能力。在划分阶段,DiCo分析输入掩码序列,识别种子词元并扩展为局部聚类;在征服阶段,对各聚类并行解码;分治过程反复交替直至收敛。最后阶段,采用细粒度复合解码方案完成剩余少量词元的生成。大量实验表明,DiCo在保持竞争力生成质量的同时,实现显著的推理加速。
原文摘要 · Abstract (English)
Diffusion-based large language models (dLLMs) have shown promising performance across various reasoning tasks, establishing themselves as an alternative to autoregressive large language models (LLMs). Unlike autoregressive LLMs that generate one token per step based on all previous tokens, dLLMs theoretically enable parallel generation of multiple tokens at each decoding step. However, recent dLLMs still favor one-token-per-step generation in practice, as directly decoding multiple masked tokens often leads to degraded generation quality and stability. This reveals a substantial gap between the theoretical parallelism and practical performance of dLLMs. To bridge this gap, we introduce an adaptive parallel decoding approach, namely DiCo, which features a three-phase divide-and-conquer paradigm to unleash the inherent parallelism of dLLMs. During the Divide phase, DiCo first explores the input masked sequence and identifies masked tokens as seed tokens, which are then expanded to construct a set of local clusters. During the Conquer phase, DiCo performs parallel decoding across different local clusters constructed in the Divide phase. The divide-and-conquer process repeatedly alternates between the Divide and Conquer phases until convergence. During the Finalize phase, DiCo decodes the remaining few masked tokens using an effective fine-grained compound decoding scheme to finalize the generation. Extensive experiments demonstrate that DiCo can achieve significant inference speedups while maintaining competitive generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。