提出可修正生成错误的离散扩散语言模型，突破传统自回归局限。
Generalized Interpolating Discrete Diffusion
- 推广掩码扩散，设计可灵活调节的噪声过程GIDD
- 在计算量相当下达到扩散语言建模最先进性能
- 混合掩码与均匀噪声，实现自我纠错能力
尽管当前顶尖语言模型通过逐词预测取得优异效果，但存在无法修改已生成词的固有缺陷。这促使人们探索替代方法，如离散扩散模型。然而，主流的掩码扩散虽简便有效，仍无法修正已有输出。为此，本文提出广义插值离散扩散（GIDD），拓展了噪声过程的设计灵活性。基于新提出的扩散变分下界（ELBO），在计算开销相当的前提下实现了扩散语言建模的最新最佳性能。利用GIDD的灵活性，进一步探索掩码与均匀噪声结合的混合策略，显著提升生成质量，并使模型具备自我修正能力，解决了自回归模型长期存在的纠错难题。
原文摘要 · Abstract (English)
While state-of-the-art language models achieve impressive results through next-token prediction, they have inherent limitations such as the inability to revise already generated tokens. This has prompted exploration of alternative approaches such as discrete diffusion. However, masked diffusion, which has emerged as a popular choice due to its simplicity and effectiveness, reintroduces this inability to revise words. To overcome this, we generalize masked diffusion, deriving a new family of general interpolating discrete diffusion (GIDD) which offers greater flexibility in the design of the noising processes. Leveraging a novel diffusion ELBO, we achieve compute-matched state-of-the-art performance in diffusion language modeling. Exploiting GIDD's flexibility, we explore a hybrid approach combining masking and uniform noise, leading to improved sample quality and unlocking the ability for the model to correct its own mistakes, an area where autoregressive models notoriously have struggled. Code: https://github.com/dvruette/gidd/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。