ReFusion让扩散模型并行生成更高效，速度超旧方法18倍且不丢质量。
ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding
- 把并行解码从词元升到槽位层级，结合扩散选择与自回归填充。
- 在7个基准上比之前扩散模型快18倍以上，性能提升34%。
- 适合追求高速高质量文本生成的研究者或工程应用。
自回归模型（ARMs）因逐词生成导致推理缓慢。虽然掩码扩散模型（MDMs）可并行生成，但存在两大缺陷：无法使用键值缓存导致计算开销高，以及在不可行的词元组合空间中学习依赖关系造成生成不连贯。为此，我们提出 extsc{ReFusion}，一种将序列重排融入因果注意力框架的新型掩码扩散模型。通过将并行解码从词元级提升至更高层级的槽位级， extsc{ReFusion} 在每次迭代中交错执行槽间扩散选择与槽内自回归补全，并在每轮后将新生成的槽位提前至待填掩码前。该设计同时实现完整键值缓存复用，并将学习复杂度从难以处理的词元组合空间降低至可控的槽位排列空间。在七个不同基准上的实验表明， extsc{ReFusion} 不仅在平均性能上相比先前的MDMs提升34%，推理速度提升超过18×，还缩小了与强自回归模型的性能差距，同时保持2.33×的平均加速优势。
原文摘要 · Abstract (English)
Autoregressive models (ARMs) are hindered by slow sequential inference. While masked diffusion models (MDMs) offer a parallel alternative, they suffer from critical drawbacks: high computational overhead from precluding Key-Value (KV) caching, and incoherent generation arising from learning dependencies over an intractable space of token combinations. To address these limitations, we introduce \textsc{ReFusion}, a novel masked diffusion model that integrates sequence reorganization into the causal attention framework. By elevating parallel decoding from the token level to a higher slot level, \textsc{ReFusion} interleaves inter-slot diffusion-based selection with intra-slot autoregressive infilling, while reordering newly generated slots ahead of the remaining masks after each iteration. Consequently, this design simultaneously unlocks full KV cache reuse and reduces learning complexity from an intractable token combination space to a manageable slot-level permutation space. Extensive experiments on seven diverse benchmarks show that \textsc{ReFusion} not only overwhelmingly surpasses prior MDMs with a 34\% performance gain and an over 18$\times$ speedup on average, but also bridges the performance gap to strong ARMs while maintaining a 2.33$\times$ average speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。