让扩散语言模型自适应生成长度，速度提升30倍
Diffusion LLM with Native Variable Generation Lengths: Let [EOS] Lead the Way
- 训练模型直接预测[EOS]标记，实现动态生成长度
- 相比传统扩散模型快30.1倍，比自回归模型快2.4倍
- 适合追求高速生成的实时应用开发者
基于扩散的大型语言模型（dLLMs）展现出并行文本生成的巨大潜力，可能比自回归模型更高效。然而，当前dLLMs存在生成长度固定的问题，即生成长度需在解码前作为超参数确定，导致效率与灵活性受限。为此，本文提出训练具有原生可变生成长度的扩散语言模型（dLLM-Var）。具体而言，我们训练模型准确预测生成文本中的[EOS]标记，使dLLM能以块扩散方式自然推断，同时保持全局双向（全）注意力和高并行性。在标准基准上的实验表明，该方法相较传统dLLM推理范式提速30.1倍，相对于Qwen和Llama等自回归模型提速2.4倍。本方法在准确率和推理速度上均表现更优，推动dLLMs从学术概念走向实际应用。代码与模型已开源。
原文摘要 · Abstract (English)
Diffusion-based large language models (dLLMs) have exhibited substantial potential for parallel text generation, which may enable more efficient generation compared to autoregressive models. However, current dLLMs suffer from fixed generation lengths, which indicates the generation lengths of dLLMs have to be determined before decoding as a hyper-parameter, leading to issues in efficiency and flexibility. To solve these problems, in this work, we propose to train a diffusion LLM with native variable generation lengths, abbreviated as dLLM-Var. Concretely, we aim to train a model to accurately predict the [EOS] token in the generated text, which makes a dLLM be able to natively infer in a block diffusion manner, while still maintaining the ability of global bi-directional (full) attention and high parallelism. Experiments on standard benchmarks demonstrate that our method achieves a 30.1x speedup over traditional dLLM inference paradigms and a 2.4x speedup relative to autoregressive models such as Qwen and Llama. Our method achieves higher accuracy and faster inference, elevating dLLMs beyond mere academic novelty and supporting their practical use in real-world applications. Codes and models have been released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。