提出块扩散模型，融合自回归与扩散优势，支持任意长度生成。
Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
- 将扩散模型分块处理，结合自回归与扩散特性。
- 实现灵活长度生成，推理效率提升，超越现有扩散模型表现。
- 适合需要高效、可控文本生成的研究者与开发者。
扩散语言模型因可并行生成和可控性优于自回归模型，但在似然建模上表现较差，且仅支持固定长度生成。本文提出一类块扩散语言模型，介于离散去噪扩散与自回归模型之间。该模型通过支持灵活长度生成，并利用键值缓存与并行采样提升推理效率，克服了两类方法的缺陷。我们提出构建有效块扩散模型的完整方案，包括高效的训练算法、梯度方差估计器及数据驱动的噪声调度以最小化方差。块扩散在语言建模基准上达到扩散模型新最优性能，并支持任意长度序列生成。代码、模型权重及项目博客已公开于 https://m-arriola.com/bd3lms。
原文摘要 · Abstract (English)
Diffusion language models offer unique benefits over autoregressive models due to their potential for parallelized generation and controllability, yet they lag in likelihood modeling and are limited to fixed-length generation. In this work, we introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models. Block diffusion overcomes key limitations of both approaches by supporting flexible-length generation and improving inference efficiency with KV caching and parallel token sampling. We propose a recipe for building effective block diffusion models that includes an efficient training algorithm, estimators of gradient variance, and data-driven noise schedules to minimize the variance. Block diffusion sets a new state-of-the-art performance among diffusion models on language modeling benchmarks and enables generation of arbitrary-length sequences. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/bd3lms
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。