0.7B连续扩散模型可仅用一步生成正确代码,效率远超传统方法。
Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or One

- 用预训练模型做双向去噪,将连续嵌入空间的扩散过程蒸馏为少步甚至单步
- 16步学生模型在HumanEval上达31.78分,超越512步教师模型
- 单步生成在HumanEval上实现7.07分,产出功能正确的程序,适合高效编码场景
语言生成通常为串行过程:自回归模型逐词输出,扩散语言模型则通过长轨迹迭代优化。本文提出PlaidQ,一个0.7B的连续扩散语言模型用于代码生成,证明其轨迹可被激进地蒸馏为少数去噪步骤——甚至仅需一步,实现高效生成。PlaidQ将预训练自回归模型改造为连续词嵌入空间的双向去噪器,采用分布匹配进行多步蒸馏,配对轨迹监督实现单步蒸馏。在相同模型规模下,PlaidQ与离散扩散语言模型在代码生成上相当。蒸馏后,16步学生模型在HumanEval和MBPP+上分别达到31.78和40.49的pass@10,超过同规模教师模型512步采样结果。极端情况下,配对轨迹蒸馏实现单步7.07 pass@1,生成功能性正确程序。结果表明连续扩散是实现少步乃至单步代码生成的可行路径。更广泛而言,连续扩散不仅是语言表示的新形式,更提供接口,使语言模型可继承连续扩散建模中的加速与蒸馏机制。训练与推理代码及模型检查点已开源。
原文摘要 · Abstract (English)
Language generation is almost universally treated as a sequential process: autoregressive models emit one token at a time, while diffusion language models replace token-level seriality with a long trajectory of iterative refinement. In this work, we introduce PlaidQ, a 0.7B continuous diffusion language model for code generation, and show that its trajectory can be aggressively distilled into only a few denoising steps---or even one, enabling efficient code generation. PlaidQ repurposes a pretrained autoregressive model as a bidirectional denoiser over continuous token embeddings. We distill PlaidQ with distribution matching for few-step generation and paired-trajectory supervision for one-step generation. At matched model scale, PlaidQ is competitive with discrete diffusion language models on code generation. Distillation then shifts the quality--compute frontier: a 16-step student reaches 31.78 and 40.49 pass@10 on HumanEval and MBPP+, surpassing the same PlaidQ teacher sampled for 512 steps. At the extreme, paired-trajectory distillation achieves 7.07 pass@1 on HumanEval with a single denoising step, producing functionally correct programs. Together, these results establish continuous diffusion as a viable path to few-step and one-step code generation. Broadly, continuous diffusion is not merely another representation for language: it provides an interface through which language models can inherit the acceleration and distillation machinery of continuous diffusion modeling. Training and inference code and model checkpoints are available at https://github.com/pengzhangzhi/plaidq.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。