用扩散模型生成代码，效果超越自回归模型。
Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model
- 采用块扩散机制与持续预训练，提升代码建模能力。
- 在相同数据和架构下，性能优于自回归基线模型。
- 适合需要编辑、推理或低资源语言的开发者使用。
基于扩散的语言模型（DLLMs）相比自回归（AR）模型具有非序列化、分块生成和更丰富的数据复用优势，但现有代码类DLLMs在相同预算下仍落后于强自回归基线。本文通过受控实验重新审视该问题，提出Stable-DiffCoder，一种基于Seed-Coder架构、数据和训练流程的分块扩散代码模型。为实现高效知识学习与稳定训练，引入增强型分块扩散持续预训练（CPT）阶段，结合定制化预热策略与分块裁剪噪声调度。在相同数据与架构条件下，Stable-DiffCoder在广泛代码基准测试中整体表现优于其自回归对应模型。仅依赖CPT与监督微调阶段，其性能超过多种约80亿参数的AR模型与DLLM，证明扩散训练可单独提升代码建模质量。此外，扩散模型的任意顺序建模有助于结构化代码的编辑与推理，并通过数据增强显著提升低资源编程语言的表现。
原文摘要 · Abstract (English)
Diffusion-based language models (DLLMs) offer non-sequential, block-wise generation and richer data reuse compared to autoregressive (AR) models, but existing code DLLMs still lag behind strong AR baselines under comparable budgets. We revisit this setting in a controlled study and introduce Stable-DiffCoder, a block diffusion code model that reuses the Seed-Coder architecture, data, and training pipeline. To enable efficient knowledge learning and stable training, we incorporate a block diffusion continual pretraining (CPT) stage enhanced by a tailored warmup and block-wise clipped noise schedule. Under the same data and architecture, Stable-DiffCoder overall outperforms its AR counterpart on a broad suite of code benchmarks. Moreover, relying only on the CPT and supervised fine-tuning stages, Stable-DiffCoder achieves stronger performance than a wide range of \~8B ARs and DLLMs, demonstrating that diffusion-based training can improve code modeling quality beyond AR training alone. Moreover, diffusion-based any-order modeling improves structured code modeling for editing and reasoning, and through data augmentation, benefits low-resource coding languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。