提出DiffuCoder模型,用扩散模型提升代码生成质量与多样性。
DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation
- 设计耦合-GRPO采样策略,优化扩散模型的强化学习训练。
- 实验显示性能提升4.4%(EvalPlus),减少对自回归偏见依赖。
- 揭示扩散模型可自主调节生成因果性,支持更灵活的代码生成。
扩散大语言模型(dLLMs)因其全局规划和迭代精炼特性,是代码生成的有力候选方案。然而,其在编码领域的训练与推理机制仍不成熟。本文系统研究了扩散模型的去噪过程及强化学习方法,训练了一个70亿参数的dLLM——DiffuCoder,基于1300亿个代码标记。分析发现:(1)扩散模型可在无需半自回归解码的情况下自主决定生成的因果性;(2)提高采样温度不仅扩大了词汇选择范围,还改变了生成顺序,形成更丰富的强化学习探索空间。为此,我们提出新型采样方法耦合-GRPO,通过构建互补掩码噪声提升训练效率并降低日志似然估计方差。实验表明,该方法使DiffuCoder在EvalPlus基准上性能提升4.4%,显著减少对自回归偏见的依赖。本工作深化了对扩散模型生成机制的理解,并提供了一套原生扩散的强化学习训练框架。
原文摘要 · Abstract (English)
Diffusion large language models (dLLMs) are compelling alternatives to autoregressive (AR) models because their denoising models operate over the entire sequence. The global planning and iterative refinement features of dLLMs are particularly useful for code generation. However, current training and inference mechanisms for dLLMs in coding are still under-explored. To demystify the decoding behavior of dLLMs and unlock their potential for coding, we systematically investigate their denoising processes and reinforcement learning (RL) methods. We train a 7B dLLM, \textbf{DiffuCoder}, on 130B tokens of code. Using this model as a testbed, we analyze its decoding behavior, revealing how it differs from that of AR models: (1) dLLMs can decide how causal their generation should be without relying on semi-AR decoding, and (2) increasing the sampling temperature diversifies not only token choices but also their generation order. This diversity creates a rich search space for RL rollouts. For RL training, to reduce the variance of token log-likelihood estimates and maintain training efficiency, we propose \textbf{coupled-GRPO}, a novel sampling scheme that constructs complementary mask noise for completions used in training. In our experiments, coupled-GRPO significantly improves DiffuCoder's performance on code generation benchmarks (+4.4\% on EvalPlus) and reduces reliance on AR bias during decoding. Our work provides deeper insight into the machinery of dLLM generation and offers an effective, diffusion-native RL training framework. https://github.com/apple/ml-diffucoder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。