对比发现扩散语言模型实际效率不如自回归模型，评估方法需改进。
How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices
- 系统分析扩散模型效率评估问题，指出现有方法存在偏差。
- 实测显示自回归模型吞吐量更高，扩散模型始终落后。
- 加速策略仅在小批量时有效，规模扩大后优势消失。
扩散语言模型（DLMs）作为自回归（AR）范式的潜在替代方案，因其可并行解码的特性被认为更具效率潜力。然而，当前开源DLMs在实际速度上常落后于AR模型，限制了其应用价值。本文通过实验基准测试与理论分析，系统研究了DLM效率问题，发现AR模型普遍具有更高吞吐量，而DLMs则持续处于劣势。我们还考察了多种加速策略，发现双缓存（dual cache）和并行解码等技术主要在小批量场景下带来收益，且随规模增长效益递减。结果表明，亟需建立更可靠的评估方法和更有效的加速机制，以推动DLM研究发展。
原文摘要 · Abstract (English)
Diffusion language models (DLMs) have emerged as a promising alternative to the long-dominant autoregressive (AR) paradigm, offering a parallelable decoding process that could yield greater efficiency. Yet, in practice, current open-source DLMs often underperform their AR counterparts in speed, limiting their real-world utility. This work presents a systematic study of DLM efficiency, identifying key issues in prior evaluation methods. Through empirical benchmarking and a theoretical analysis, we demonstrate that AR models generally achieve higher throughput, while DLMs consistently lag. We also investigate acceleration strategies, finding that techniques like dual cache and parallel decoding mainly offer gains at small batch sizes, with their benefits diminishing upon scaling. Our findings underscore the necessity of robust evaluation methods and improved acceleration strategies to advance research on DLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。