统一自回归与扩散模型，让文本生成更灵活高效。
Unifying Autoregressive and Diffusion-Based Sequence Generation
- 用位置专属噪声调度，融合自回归与扩散模型优点。
- 新推理算法支持修正历史错误，生成质量显著提升。
- 兼容缓存机制，适合大规模文本生成任务。
我们对基于扩散的序列生成模型进行了重要扩展，使其与自回归语言模型的界限更加模糊。提出超调度（hyperschedules），为每个词元位置分配独立的噪声调度，使自回归模型（如GPT）和传统扩散模型（如SEDD、MDLM）成为其特例。其次，设计两种混合词元级加噪过程，在吸收与均匀过程间插值，使模型能修正先前错误；并引入一种新颖的推理算法，借鉴MDLM简化上下文。为支持高效训练与推理，设计了兼容KV缓存的注意力掩码。在标准基准上达到最先进困惑度，生成多样且高质量的序列，表明自回归与扩散结合路径具有巨大潜力。代码与资源见：https://hdlm-colm.github.io/
原文摘要 · Abstract (English)
We present significant extensions to diffusion-based sequence generation models, blurring the line with autoregressive language models. We introduce hyperschedules, which assign distinct noise schedules to individual token positions, generalizing both autoregressive models (e.g., GPT) and conventional diffusion models (e.g., SEDD, MDLM) as special cases. Second, we propose two hybrid token-wise noising processes that interpolate between absorbing and uniform processes, enabling the model to fix past mistakes, and we introduce a novel inference algorithm that leverages this new feature in a simplified context inspired from MDLM. To support efficient training and inference, we design attention masks compatible with KV-caching. Our methods achieve state-of-the-art perplexity and generate diverse, high-quality sequences across standard benchmarks, suggesting a promising path for autoregressive diffusion-based sequence generation. See code and resources at https://hdlm-colm.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。