开源70亿参数代码生成模型,能自适应选择生成策略。
Dream-Coder 7B: An Open Diffusion Language Model for Code
- 用扩散模型框架替代传统逐词生成,可灵活选择先画草图、顺序生成或边思考边写。
- 在LiveCodeBench上达21.4%通过率,多项基准测试表现优异。
- 适合研究代码生成与非自回归模型的开发者,开源全链条工具支持复现。
我们提出Dream-Coder 7B,一个用于代码生成的开源离散扩散语言模型,具备涌现的任意顺序生成能力。不同于传统自回归(AR)模型严格从左到右解码,Dream-Coder 7B根据任务动态选择策略:复杂算法采用草图先行生成,简单补全使用左到右生成,代码理解任务则采用交织推理生成。我们通过连续时间加权交叉熵目标,将预训练自回归检查点迁移至离散扩散框架。后训练流程包含(i)监督微调,通过随机截断和填充惩罚缓解填充病态性,提升样本效率与生成稳定性;(ii)基于高质量提示集的可验证奖励强化学习,采用专为扩散语言模型设计的强化学习方案。最终模型Dream-Coder-7B Instruct在LiveCodeBench(2410–2505)上达到21.4% pass@1,且在HumanEval、MBPP、BigCodeBench和CRUXEval上表现具有竞争力。我们发布Dream-Coder-7B及Dream-Coder-7B-Instruct模型检查点、训练方案、预处理管道和推理代码,以促进可复现性和进一步研究。
原文摘要 · Abstract (English)
We present Dream-Coder 7B, an open-source discrete diffusion language model for code generation that exhibits emergent any-order generation capabilities. Unlike traditional autoregressive (AR) models that decode strictly left-to-right, Dream-Coder 7B adaptively determines its decoding strategy based on the coding task: sketch-first generation for complex algorithms, left-to-right generation for straightforward completions, and interleaved reasoning generation for code understanding tasks. We adapt a pretrained AR checkpoint to a discrete diffusion frameworks with a continuous-time weighted cross-entropy objective. Our post-training recipe comprises (i) supervised fine-tuning, where we mitigate padding pathologies via random truncation and a padding penalty to improve sample efficiency and stabilize generation; and (ii) reinforcement learning with verifiable rewards over a curated high-quality prompt set drawn from open-source datasets, using a tailored reinforcement learning recipe for diffusion language models. The resulting Dream-Coder 7B Instruct attains 21.4\% pass@1 on LiveCodeBench (2410--2505) and demonstrates competitive performance on HumanEval, MBPP, BigCodeBench, and CRUXEval. We release Dream-Coder-7B and Dream-Coder-7B-Instruct checkpoints, training recipes, preprocessing pipelines, and inference code to facilitate reproducibility and further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。