系统梳理并对比并行文本生成技术,提升大模型生成速度。
A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models
- 按自回归与非自回归范式分类,梳理主流并行生成方法。
- 揭示不同方法在速度、质量、效率间的权衡关系。
- 适合关注大模型推理加速的研究者与工程实践者。
随着文本生成成为现代大语言模型的核心能力,其支撑了众多下游应用。然而,现有大多数大模型依赖自回归生成,逐个生成词元,受限于过程的固有串行性,生成速度有限。为应对这一挑战,越来越多研究者开始探索并行文本生成——一类旨在打破逐词生成瓶颈、提升推理效率的技术。尽管兴趣日益增长,但对并行生成技术的具体构成及其如何提升推理性能仍缺乏系统分析。为此,本文提出一项系统性综述,将现有方法分为基于自回归与非自回归两大范式,并详细剖析各类核心技术。在此分类基础上,评估其在速度、质量与效率上的理论权衡,并探讨其与其它加速策略的结合潜力。最后,基于发现,总结最新进展,识别开放挑战,并展望未来研究方向。相关论文与资源已整理至 GitHub 仓库:https://github.com/zhanglingzhe0820/Awesome-Parallel-Text-Generation。
原文摘要 · Abstract (English)
As text generation has become a core capability of modern Large Language Models (LLMs), it underpins a wide range of downstream applications. However, most existing LLMs rely on autoregressive (AR) generation, producing one token at a time based on previously generated context-resulting in limited generation speed due to the inherently sequential nature of the process. To address this challenge, an increasing number of researchers have begun exploring parallel text generation-a broad class of techniques aimed at breaking the token-by-token generation bottleneck and improving inference efficiency. Despite growing interest, there remains a lack of comprehensive analysis on what specific techniques constitute parallel text generation and how they improve inference performance. To bridge this gap, we present a systematic survey of parallel text generation methods. We categorize existing approaches into AR-based and Non-AR-based paradigms, and provide a detailed examination of the core techniques within each category. Following this taxonomy, we assess their theoretical trade-offs in terms of speed, quality, and efficiency, and examine their potential for combination and comparison with alternative acceleration strategies. Finally, based on our findings, we highlight recent advancements, identify open challenges, and outline promising directions for future research in parallel text generation. We have also created a GitHub repository for indexing relevant papers and open resources available at https://github.com/zhanglingzhe0820/Awesome-Parallel-Text-Generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。