arXiv:2606.01024cs.CLcs.AI2026-06

让大模型用连续去噪生成文本,更长更准不重复。

DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMs

论文配图:DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMs
图 1 · 摘自论文原文
  • 用高斯噪声替代二值掩码,轻量微调实现连续去噪。
  • 16步内生成长度翻倍且无重复,ROUGE-1全超基准。
  • 能修复错字又保原词,适合低资源生成场景。

离散掩码扩散语言模型通过迭代并行解码生成文本,但少步数解码存在长度与质量的权衡:固定步数下,传统方法只能生成短而高质量的输出,或生成长但重复的内容。连续去噪可通过联合演化嵌入空间中所有位置来规避这一权衡,但大规模构建此类模型仍具挑战。本文表明,可对预训练的掩码语言模型进行轻量级适配以支持连续嵌入空间去噪。基于LLaDA-8B-Instruct,仅用1,000步继续预训练,采用离散随机定位(DSL),将二值掩码替换为每标记的连续高斯噪声作为软掩码。所获模型支持连续推理,在嵌入空间中联合演化所有位置,并将硬词选择推迟至最后一步。在零样本摘要任务中,低步数预算(≤16次前向传播)下,DSL-LLaDA-SDE在四个基准上均取得最佳ROUGE-1,显著避免了迭代解码中的过早终止与重复问题。相同适配也带来对噪声状态的选择性鲁棒性:模型可修正受损词元同时保留干净词元。控制实验显示,使用相同算力的标准掩码扩散训练无法获得上述行为。

原文摘要 · Abstract (English)

Discrete Masked diffusion language models generate text by iterative parallel decoding, but few-step decoding suffers from a tradeoff between length and quality: with a fixed step budget, standard methods can generate a short, high-quality output, or they can produce long but repetitive text. Continuous denoising can sidestep this tradeoff by evolving all positions jointly in embedding space, but building such a model from scratch at scale remains an open problem. We show that a pretrained masked DLM can instead be lightly adapted to support continuous embedding-space denoising. Starting from LLaDA-8B-Instruct, we continue-pretrain for only 1,000 steps with Discrete Stochastic Localization (DSL), replacing binary masking with continuous per-token Gaussian noise as a soft mask. The adapted model supports continuous inference that evolves all positions jointly in embedding space and defers hard token commitment to the final step. On zero-shot summarization at low step budgets (<=16 forward passes), DSL-LLaDA-SDE achieves the best ROUGE-1 on all four benchmarks and largely avoids the premature-termination / repetition tradeoff of iterative unmasking. The same adaptation also yields selective noisy-state robustness: the model corrects corrupted tokens while preserving clean ones. Control experiments using standard masked diffusion training with the same compute demonstrate neither behavior.

连续去噪大模型生成文本生成扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。