用扩散模型并行生成文本，实现零质量损失的加速。
Unlocking Lossless Speedups in LLMs via Discrete Diffusion
- 将自回归模型与轻量扩散权重解耦，通过蒸馏训练并行采样。
- 在不牺牲质量前提下，最高达3倍的生成速度提升。
- 无需额外草稿模型，适合需要高效推理的应用场景。
大型语言模型（LLMs）的成功很大程度上依赖于下一个词预测（NTP），但其自回归（AR）结构导致生成过程缓慢且串行。为突破这一瓶颈，我们提出扩散增强型LLM，该模型在保持标准自回归分布的同时，利用扩散机制从该分布中并行生成多个词元。模型参数被解耦为两部分：通过标准NTP目标训练的自回归权重，以及通过简单扩散蒸馏阶段训练的轻量级扩散权重，该阶段对现有训练流程开销极小。我们还引入Ψ-Spec系列采样器，在固定上下文长度下实现无损加速与推理时缩放。与推测解码不同，本方法无需独立草稿模型；与扩散LLM（d-LLMs）相比，它在不降低基础自回归模型质量的前提下实现生成加速。所提出的模型名为Uno，可从头训练或基于现有开源自回归模型构建。实验显示，Uno在所有评估批次大小下均优于领先推测解码方法，最大速度提升达3倍，且在设备支持的最大批大小下仍保持优势。值得注意的是，8B版本的Uno在代理工具使用、编程和长上下文推理等任务中表现优于26B的DiffusionGemma及专有模型Mercury 2。代码与模型权重已发布：https://s-sahoo.github.io/uno/
原文摘要 · Abstract (English)
Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation. To overcome this bottleneck, we introduce diffusion-augmented LLMs, a new class of models that defines an AR model distribution while using diffusion to draw multiple tokens in parallel from that distribution. We decouple the parameters of these models into two sets: AR weights, trained using the standard NTP objective, and lightweight diffusion weights, trained to generate multiple tokens simultaneously. The diffusion weights are learned through a simple Diffusion Distillation phase that adds negligible overhead to existing LLM training pipelines. We also introduce $\Psi$-Spec, a family of samplers that enables lossless acceleration and inference-time scaling at a fixed context length. Unlike speculative decoding, our method requires no separate draft model. Unlike diffusion LLMs (d-LLMs), it accelerates generation without sacrificing the quality of the underlying AR model. The resulting models, called Uno, can be trained from scratch or built by augmenting existing open-weight AR LLMs. Uno achieves higher throughput than leading speculative-decoding methods at every evaluated batch size and delivers up to $3\times$ speedups over the base AR model, including at the largest batch size supported by the device. Notably, our 8B Uno model outperforms the leading open d-LLM, the 26B DiffusionGemma, and the proprietary Mercury 2 across all evaluated benchmarks in agentic tool use, coding, and long-context reasoning. We release code and checkpoints at: https://s-sahoo.github.io/uno/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。