首次实证研究扩散模型在代码生成中的表现,发现其效率与长代码理解能力更优。
Beyond Autoregression: An Empirical Study of Diffusion Large Language Models for Code Generation
- 采用多标记预测和灵活生成顺序,突破传统逐词生成的限制。
- 在4个基准测试中,扩散模型性能与同规模自回归模型相当,长代码生成更优。
- 揭示影响效率的关键因素,为后续研究提供实用指导。
大语言模型已成为代码生成的主流方法,主要采用自回归生成方式,即从左到右逐个生成代码标记。然而该方式存在两大局限:一是每步仅生成一个标记,实际效率低;二是编程是反复修改的非顺序过程,而自回归模型固定左到右顺序。近期扩散大语言模型成为有前景的替代方案,通过多标记预测和灵活生成顺序解决上述问题。但目前尚无系统性研究探索其在代码生成中的应用。为此,本文首次开展扩散大语言模型在代码生成上的实证研究,涵盖9个代表性扩散模型,在4个常用基准上进行实验。结果表明:(1)现有扩散模型在相同规模下性能可媲美自回归模型;(2)扩散模型具备更强的长度外推能力,在长代码理解任务中表现更优;(3)分析影响其效率与效果的关键因素,并给出实践建议;(4)提出若干改进方向。所有代码、数据与结果已开源,地址为 https://github.com/zhangyitonggg/dllm4code。
原文摘要 · Abstract (English)
LLMs have become the mainstream approaches to code generation. Existing LLMs mainly employ autoregressive generation, i.e. generating code token-by-token from left to right. However, the underlying autoregressive generation has two limitations in code generation. First, autoregressive LLMs only generate a token at each step, showing low efficiency in practice. Second, programming is a non-sequential process involving back-and-forth editing, while autoregressive LLMs only employ the left-to-right generation order. These two intrinsic limitations hinder the further development of LLMs in code generation. Recently, diffusion LLMs have emerged as a promising alternative. Diffusion LLMs address the above limitations with two advances, including multi-token prediction (i.e. generating multiple tokens at each step) and flexible generation order (i.e. flexibly determining which positions to generate tokens). However, there is no systematic study exploring diffusion LLMs in code generation. To bridge the knowledge gap, we present the first empirical study of diffusion LLMs for code generation. Our study involves 9 representative diffusion LLMs and conduct experiments on 4 widely used benchmarks. Based on the results, we summarize the following findings. (1) Existing diffusion LLMs are competitive with autoregressive LLMs with similar sizes. (2) Diffusion LLMs have a stronger length extrapolation ability than autoregressive LLMs and perform better in long code understanding. (3) We explore factors impacting the effectiveness and efficiency of diffusion LLMs, and provide practical guidance. (4) We discuss several promising further directions to improve diffusion LLMs on code generation. We open-source all source code, data, and results to facilitate the following research. The code is publicly available at https://github.com/zhangyitonggg/dllm4code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。