新模型通过灵活排列词元实现快速解码，兼顾生成质量与推理效率。
Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding
- 基于可变位置、可变长度的词元集合进行概率建模，突破固定块限制。
- 每步推理后更新键值缓存，支持任意顺序解码与滑动窗口生成。
- 在数学推理等任务中比现有扩散模型更快更准，适合高效生成场景。
离散扩散模型在生成质量上已接近自回归（AR）模型，但通常受限于固定长度生成且不支持键值（KV）缓存。块扩散通过左到右生成词元块部分弥合了两者差距，但固定大小的序列块限制了解码灵活性与并行性。本文提出一种新型语言模型——集合扩散（set diffusion），包含：(i) 基于灵活位置与长度词元集合的概率参数化；(ii) 支持每步推理后更新KV缓存的集合因果扩散架构。通过在词元集合而非固定块上进行因式分解，词元可任意顺序解码，包括滑动窗口形式，从而实现更快推理与任意顺序生成。在数学推理、摘要生成和无条件生成任务中，该模型相比以往扩散语言模型展现出更优的速度-质量权衡，同时在填充任务上的表现优于块扩散。代码、模型权重及项目博客已公开：https://m-arriola.com/setdlms/
原文摘要 · Abstract (English)
Discrete diffusion models have steadily improved in quality relative to autoregressive (AR) models. However, these models are normally constrained to fixed-length generation and do not support key-value (KV) caching. Block diffusion partially bridges diffusion and AR by generating token blocks left-to-right, but its fixed-size sequential blocks limit decoding flexibility and parallelism. Here, we present a new class of language models, set diffusion, comprised of (i) a likelihood parameterization that factorizes over flexible-position, flexible-length token sets and (ii) a set-causal diffusion architecture that supports KV cache updates after every inference step. By factorizing over token sets instead of fixed-size blocks, tokens can be decoded in arbitrarily-ordered sets, including sliding-window sets, enabling faster inference and support for any-order decoding. Set diffusion achieves better speed-quality tradeoffs on mathematical reasoning, summarization, and unconditional generation compared to prior diffusion language models while offering stronger infilling performance than block diffusion. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/setdlms/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。