提出d3LLM模型，让扩散语言模型更快更准。
d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation
- 用伪轨迹蒸馏教模型早期就能自信预测部分词元，提升并行生成能力。
- 推理时结合熵判据和缓存刷新，实现10倍加速且准确率损失小。
- 适合追求高速生成又不想牺牲质量的AI应用开发者。
扩散型大语言模型（dLLMs）具备并行解码和随机顺序生成等超越自回归模型的能力，但实际应用中面临准确率与并行性之间的权衡。现有方法多侧重于效率或准确率其中之一。为此，我们提出d3LLM（伪蒸馏扩散大语言模型），在训练阶段引入伪轨迹蒸馏，指导模型识别早期可高置信度解码的词元，从而提升并行性；推理阶段采用基于熵的多块解码与键值缓存刷新机制，在保持准确率的同时实现高并行性。为更全面评估dLLMs，我们还提出新指标AUP（Accuracy Under Parallelism），联合衡量准确率与并行性。实验表明，d3LLM相比原生LLaDA/Dream模型最高提速10倍，相比自回归模型提速5倍，且准确率下降不显著。代码已开源：https://github.com/hao-ai-lab/d3LLM。
原文摘要 · Abstract (English)
Diffusion large language models (dLLMs) offer capabilities beyond those of autoregressive (AR) LLMs, such as parallel decoding and random-order generation. However, realizing these benefits in practice is non-trivial, as dLLMs inherently face an accuracy-parallelism trade-off. Despite increasing interest, existing methods typically focus on only one-side of the coin, targeting either efficiency or accuracy. To address this limitation, we propose d3LLM (Pseudo-Distilled Diffusion Large Language Model), striking a balance between accuracy and parallelism: (i) during training, we introduce pseudo-trajectory distillation to teach the model which tokens can be decoded confidently at early steps, thereby improving parallelism; (ii) during inference, we employ entropy-based multi-block decoding with a KV-cache refresh mechanism to achieve high parallelism while maintaining accuracy. To better evaluate dLLMs, we also introduce AUP (Accuracy Under Parallelism), a new metric that jointly measures accuracy and parallelism. Experiments demonstrate that our d3LLM achieves up to 10$\times$ speedup over vanilla LLaDA/Dream, and 5$\times$ speedup over AR models without much accuracy drop. Our code is available at https://github.com/hao-ai-lab/d3LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。