将掩码扩散模型放入解码器架构，实现高效推理与自回归模型公平对比。
Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
- 在解码器框架中实现任意顺序的掩码扩散模型，消除架构差异干扰。
- 推理速度提升约25倍，困惑度与传统自回归模型相当。
- 适合关注推理效率优化的模型设计者和大模型开发者。
高效扩展大语言模型需要探索主流自回归（AR）方法的替代方案，掩码扩散模型（MDM）成为候选。然而，由于自回归模型通常采用解码器结构，而掩码扩散模型多为编码器结构，架构差异使两者算法与效率的真正权衡难以判断。本文通过在解码器仅有的框架下评估掩码扩散模型，实现：(1) 以任意顺序的自回归形式公平比较掩码扩散模型与标准自回归模型；(2) 探究掩码扩散模型对计算效率的影响。结果显示，尽管建模空间更大，解码器仅有的掩码扩散模型结合温度退火等技术，可实现约25倍的推理加速，同时保持与标准模型相近的困惑度，为降低推理计算成本提供新路径。本研究有助于分离核心建模选择与架构影响，推动更高效的基座模型发展。代码已公开于 https://github.com/scxue/AO-GPT-MDM。
原文摘要 · Abstract (English)
Efficiently scaling Large Language Models (LLMs) necessitates exploring alternatives to dominant autoregressive (AR) methods, with Masked Diffusion Models (MDMs) emerging as candidates. However, comparing AR (typically decoder-only) and MDM (often encoder-only) paradigms is confounded by differing architectures, obscuring true algorithmic and efficiency trade-offs. This research decouples these factors by evaluating MDMs within a decoder-only framework to: (1) Equitably compare MDM (as Any-Order AR) and standard AR paradigms through discrepancies on orders. (2) Investigate MDM architectural impacts on computational efficiency. We show decoder-only MDMs, despite a larger modeling space, can achieve significant inference speedups ($\sim25\times$) and comparable perplexity with techniques like temperature annealing, offering a path to reduced inference compute. This work provides insights for developing more computationally efficient foundation models by disentangling core modeling choices from architectural influences. Code is available at https://github.com/scxue/AO-GPT-MDM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。