让自回归模型高效适配扩散生成，实现并行加速且无需重新预训练。
From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons
- 通过严格因果对齐，直接继承GPT类模型权重，跳过海量预训练。
- 动态调节去噪步长，依据局部信息密度而非固定规则，提升生成效率。
- 适合想用现有自回归模型快速构建高效生成系统的研究者与开发者。
扩散模型虽能实现高效的并行文本生成，但依赖双向注意力机制，与预训练的自回归（AR）模型存在结构不匹配，导致无法复用成熟的AR先验，只能从头进行代价高昂的预训练。为此，我们提出FLUID框架，通过强制执行严格因果对齐，使AR骨干网络可无缝初始化于标准GPT类检查点，避免大规模预训练。此外，我们引入弹性时域（Elastic Horizons）机制，基于局部信息密度的熵驱动方式动态调节去噪步长，而非采用固定调度。实验表明，FLUID在保持领先性能的同时，将训练成本降低至原有量级的数个数量级，有效弥合了成熟自回归基础与高效并行生成之间的鸿沟。代码已公开于https://github.com/Oli-lab-nun/FLUID/tree/main。
原文摘要 · Abstract (English)
Diffusion models promise efficient parallel text generation but rely on bidirectional attention, creating a structural mismatch with pre-trained Autoregressive (AR) models. This incompatibility precludes reusing robust AR priors, necessitating prohibitive pre-training from scratch. To bridge this gap, we propose FLUID, a framework that efficiently adapts AR backbones to the diffusion paradigm. By enforcing Strictly Causal Alignment, FLUID enables seamless initialization from standard GPT-style checkpoints, circumventing the need for massive pre-training. Furthermore, we introduce Elastic Horizons, an entropy-driven mechanism that dynamically modulates denoising strides based on local information density rather than fixed schedules. Experiments demonstrate that FLUID achieves state-of-the-art performance while reducing training costs by orders of magnitude, effectively reconciling established AR foundations with efficient parallel generation. Our code is available at https://github.com/Oli-lab-nun/FLUID/tree/main.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。