提出新型扩散语言模型，兼顾生成速度与质量，支持缓存加速推理。
Esoteric Language Models: A Family of Any-Order Diffusion LLMs
- 融合自回归与扩散思想，用因果注意力实现精确似然计算。
- 首次在并行生成中引入KV缓存，推理速度显著提升。
- 在无条件生成上达到新最优平衡，适合高效生成场景。
基于扩散的语言模型为自回归（AR）模型提供了有力替代，支持并行和可控生成。当前表现最佳的掩码扩散模型（MDMs）在困惑度上仍逊于自回归模型，且缺乏关键的推理效率特性，如键值缓存（KV caching）。本文提出Eso-LMs，一个融合AR与MDM范式的新型模型家族，可在两者之间平滑插值，同时克服各自局限。不同于以往使用双向注意力的MDM去噪器，我们利用MDM与任意顺序自回归模型之间的联系，采用因果注意力机制。这一设计首次实现了对MDMs的精确似然计算，并关键性地在保持并行生成的同时引入了KV缓存，显著提升推理效率。结合优化采样调度，Eso-LMs在无条件生成任务上建立了新的速度-质量帕累托前沿。代码、模型权重及视频教程已公开于项目主页：https://s-sahoo.com/Eso-LMs。
原文摘要 · Abstract (English)
Diffusion-based language models offer a compelling alternative to autoregressive (AR) models by enabling parallel and controllable generation. Within this family, Masked Diffusion Models (MDMs) currently perform best but still underperform AR models in perplexity and lack key inference-time efficiency features, most notably KV caching. We introduce Eso-LMs, a new family of models that fuses AR and MDM paradigms, smoothly interpolating between their perplexities while overcoming their respective limitations. Unlike prior work, which uses transformers with bidirectional attention as MDM denoisers, we exploit the connection between MDMs and Any-Order autoregressive models and adopt causal attention. This design lets us compute the exact likelihood of MDMs for the first time and, crucially, enables us to introduce KV caching for MDMs while preserving parallel generation for the first time, significantly improving inference efficiency. Combined with an optimized sampling schedule, Eso-LMs establish a new state of the art on the speed-quality Pareto frontier for unconditional generation. We provide the code, model checkpoints, and the video tutorial on the project page: https://s-sahoo.com/Eso-LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。