arXiv:2509.01142cs.CL2025-09被引 76

开源70亿参数代码生成模型,能自适应选择生成策略。

Dream-Coder 7B: An Open Diffusion Language Model for Code

  • 用扩散模型框架替代传统逐词生成,可灵活选择先画草图、顺序生成或边思考边写。
  • 在LiveCodeBench上达21.4%通过率,多项基准测试表现优异。
  • 适合研究代码生成与非自回归模型的开发者,开源全链条工具支持复现。

我们提出Dream-Coder 7B,一个用于代码生成的开源离散扩散语言模型,具备涌现的任意顺序生成能力。不同于传统自回归(AR)模型严格从左到右解码,Dream-Coder 7B根据任务动态选择策略:复杂算法采用草图先行生成,简单补全使用左到右生成,代码理解任务则采用交织推理生成。我们通过连续时间加权交叉熵目标,将预训练自回归检查点迁移至离散扩散框架。后训练流程包含(i)监督微调,通过随机截断和填充惩罚缓解填充病态性,提升样本效率与生成稳定性;(ii)基于高质量提示集的可验证奖励强化学习,采用专为扩散语言模型设计的强化学习方案。最终模型Dream-Coder-7B Instruct在LiveCodeBench(2410–2505)上达到21.4% pass@1,且在HumanEval、MBPP、BigCodeBench和CRUXEval上表现具有竞争力。我们发布Dream-Coder-7B及Dream-Coder-7B-Instruct模型检查点、训练方案、预处理管道和推理代码,以促进可复现性和进一步研究。

原文摘要 · Abstract (English)

We present Dream-Coder 7B, an open-source discrete diffusion language model for code generation that exhibits emergent any-order generation capabilities. Unlike traditional autoregressive (AR) models that decode strictly left-to-right, Dream-Coder 7B adaptively determines its decoding strategy based on the coding task: sketch-first generation for complex algorithms, left-to-right generation for straightforward completions, and interleaved reasoning generation for code understanding tasks. We adapt a pretrained AR checkpoint to a discrete diffusion frameworks with a continuous-time weighted cross-entropy objective. Our post-training recipe comprises (i) supervised fine-tuning, where we mitigate padding pathologies via random truncation and a padding penalty to improve sample efficiency and stabilize generation; and (ii) reinforcement learning with verifiable rewards over a curated high-quality prompt set drawn from open-source datasets, using a tailored reinforcement learning recipe for diffusion language models. The resulting Dream-Coder 7B Instruct attains 21.4\% pass@1 on LiveCodeBench (2410--2505) and demonstrates competitive performance on HumanEval, MBPP, BigCodeBench, and CRUXEval. We release Dream-Coder-7B and Dream-Coder-7B-Instruct checkpoints, training recipes, preprocessing pipelines, and inference code to facilitate reproducibility and further research.

代码生成扩散模型开源模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。