让扩散模型自由生成任意长度代码，突破固定长度限制。
DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas
- 引入长度控制状态，实现输出长度动态调整。
- 在HumanEval上性能媲美顶尖自回归模型，接近理想长度表现。
- 无需修改架构，轻松集成现有扩散语言模型。
扩散语言模型（DLMs）为代码补全提供了灵活的非自回归生成方式，但受限于必须预设固定长度掩码，当掩码大小与理想补全长度不匹配时性能显著下降。为此，我们提出DreamOn，一种支持动态可变长度生成的新框架。通过在扩散过程中引入两个长度控制状态，模型可自主根据自身预测扩展或收缩输出长度。该机制仅需微调训练目标，无需改变原有模型架构。基于Dream-Coder-7B和DiffuCoder-7B构建的DreamOn，在HumanEval-Infilling和SantaCoder-FIM任务上达到与当前最优自回归模型相当的性能，并接近使用真实长度标注的最优表现。本工作消除了DLM实际应用的关键障碍，大幅提升了其在可变长度生成任务中的灵活性与实用性。代码已开源。
原文摘要 · Abstract (English)
Diffusion Language Models (DLMs) present a compelling alternative to autoregressive models, offering flexible, any-order infilling without specialized prompting design. However, their practical utility is blocked by a critical limitation: the requirement of a fixed-length masked sequence for generation. This constraint severely degrades code infilling performance when the predefined mask size mismatches the ideal completion length. To address this, we propose DreamOn, a novel diffusion framework that enables dynamic, variable-length generation. DreamOn augments the diffusion process with two length control states, allowing the model to autonomously expand or contract the output length based solely on its own predictions. We integrate this mechanism into existing DLMs with minimal modifications to the training objective and no architectural changes. Built upon Dream-Coder-7B and DiffuCoder-7B, DreamOn achieves infilling performance on par with state-of-the-art autoregressive models on HumanEval-Infilling and SantaCoder-FIM and matches oracle performance achieved with ground-truth length. Our work removes a fundamental barrier to the practical deployment of DLMs, significantly advancing their flexibility and applicability for variable-length generation. Our code is available at https://github.com/DreamLM/DreamOn.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。