用自回归框架实现任意顺序生成,效果超越扩散模型。
Autoregressive Models Rival Diffusion Models at ANY-ORDER Generation
- 将自回归模型扩展为任意分组与顺序的生成框架
- 在问答、推理等任务上优于扩散模型,且支持灵活解码
- 适合需要高效、灵活生成的应用场景
扩散语言模型可实现任意顺序生成和双向条件化,适用于填空、重写和自我修正等任务。然而,其单步依赖的建模方式限制了模型深度,通常生成质量与稳定性不如自回归(AR)模型。为此,本文重新以自回归建模为基础,将扩散式训练重构为结构化的多组预测过程,提出任意顺序任意子集自回归建模(A3)。A3将标准AR分解推广至任意标记组和生成顺序,在保持AR的概率严谨性和多层依赖建模能力的同时,继承了扩散模型的并行与双向生成灵活性。通过双流注意力架构与渐进适配策略,将预训练自回归模型逐步转化为任意顺序预测。在问答、常识推理和故事补全任务上的实验表明,A3不仅性能超越基于扩散的模型,且保留灵活解码能力。本工作提供了一种统一、高效、新颖的语言建模新范式。
原文摘要 · Abstract (English)
Diffusion language models enable any-order generation and bidirectional conditioning, offering appealing flexibility for tasks such as infilling, rewriting, and self-correction. However, their formulation-predicting one part of a sequence from another within a single-step dependency-limits modeling depth and often yields lower sample quality and stability than autoregressive (AR) models. To address this, we revisit autoregressive modeling as a foundation and reformulate diffusion-style training into a structured multi-group prediction process. We propose Any-order Any-subset Autoregressive modeling (A3), a generalized framework that extends the standard AR factorization to arbitrary token groups and generation orders. A3 preserves the probabilistic rigor and multi-layer dependency modeling of AR while inheriting diffusion models' flexibility for parallel and bidirectional generation. We implement A3 through a two-stream attention architecture and a progressive adaptation strategy that transitions pretrained AR models toward any-order prediction. Experiments on question answering, commonsense reasoning, and story infilling demonstrate that A3 outperforms diffusion-based models while maintaining flexible decoding. This work offers a unified approach for a flexible, efficient, and novel language modeling paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。