用10亿词元微调实现高效并行文本生成,速度提升2.5倍。
Fast-dLLM v2: Efficient Block-Diffusion LLM
- 采用分块扩散机制与互补注意力掩码,实现块级双向上下文建模。
- 仅需约10亿词元微调,较同类模型减少500倍训练数据。
- 适合追求高速推理且不牺牲生成质量的部署场景。
自回归大语言模型在自然语言任务中表现卓越,但其固有的序列解码限制了推理效率。本文提出 Fast-dLLM v2,一种精心设计的块扩散语言模型(dLLM),可高效将预训练自回归模型转化为支持并行生成的dLLM,仅需约10亿词元的微调。相比全注意力扩散LLM如Dream(5800亿词元),训练数据量减少500倍,同时保持原始模型性能。方法引入新颖训练范式,结合块扩散机制与互补注意力掩码,实现块内双向上下文建模而不损害自回归训练目标。为加速解码,设计分层缓存机制:块级缓存存储各块历史上下文表示,子块缓存实现部分解码块内的高效并行生成。配合并行解码流水线,Fast-dLLM v2 在不降低生成质量的前提下,相较标准自回归解码实现最高2.5倍速度提升。跨多种基准的广泛实验表明,Fast-dLLM v2在准确性上达到或超越自回归基线,是当前dLLM中效率最优的方案,标志着快速、准确大模型实用化的重要进展。代码与模型将公开发布。
原文摘要 · Abstract (English)
Autoregressive (AR) large language models (LLMs) have achieved remarkable performance across a wide range of natural language tasks, yet their inherent sequential decoding limits inference efficiency. In this work, we propose Fast-dLLM v2, a carefully designed block diffusion language model (dLLM) that efficiently adapts pretrained AR models into dLLMs for parallel text generation, requiring only approximately 1B tokens of fine-tuning. This represents a 500x reduction in training data compared to full-attention diffusion LLMs such as Dream (580B tokens), while preserving the original model's performance. Our approach introduces a novel training recipe that combines a block diffusion mechanism with a complementary attention mask, enabling blockwise bidirectional context modeling without sacrificing AR training objectives. To further accelerate decoding, we design a hierarchical caching mechanism: a block-level cache that stores historical context representations across blocks, and a sub-block cache that enables efficient parallel generation within partially decoded blocks. Coupled with our parallel decoding pipeline, Fast-dLLM v2 achieves up to 2.5x speedup over standard AR decoding without compromising generation quality. Extensive experiments across diverse benchmarks demonstrate that Fast-dLLM v2 matches or surpasses AR baselines in accuracy, while delivering state-of-the-art efficiency among dLLMs - marking a significant step toward the practical deployment of fast and accurate LLMs. Code and model will be publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。