让扩散语言模型一步生成完整词块,提升效率与连贯性。
A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Models

- 通过耦合噪声蒸馏,让模型在单步内预测完整词块。
- 单步生成结果合法率显著提升,不同噪声产生不同输出。
- 适合追求高效生成的文本生成研究者使用。
自回归语言模型每次前向传播只生成一个词元;扩散语言模型则需多步逐步生成词块。本文探讨能否在单次前向传播中完成整个词块的生成。为此,采用一种噪声条件化的掩码去噪器:在掩码嵌入上添加独立于数据的高斯噪声场,理论上每个采样场可选择一个词块的联合模式。传统训练方式是为每个样本采样多个噪声场,通过胜者为王或重要性加权使它们竞争数据。但此方法对噪声的控制较粗略:实验显示,噪声所携带信息随竞争场数量的对数增长,且在多种模型规模下,单步生成结果仍很少连贯。本文提出 CONDOR(耦合噪声蒸馏用于单步读出)。先用随机掩码位置和胜者为王训练一个噪声条件化教师模型;学生模型生成单步词块,保留选中的词元,并在相同噪声场下由教师通过多步填充剩余位置获得目标块进行学习;同时使用无噪声的掩码语言模型项锚定真实标签。人类评估在 TinyStories 数据集上显示,单步合法性大幅提高,不同噪声场仍能生成不同词块,实现每块仅一次前向传播。
原文摘要 · Abstract (English)
Autoregressive language models commit one token per forward pass; diffusion language models commit a block of tokens over several steps. We ask whether a block can be committed in a single forward pass. We study this with a noise-conditioned masked denoiser: a data-independent Gaussian noise field is added to the mask embeddings so that, in principle, each sampled field selects one joint mode of the block. The established way of training such a model is to sample several fields per example and let them compete for the data, by winner-take-all or importance weighting. This gives the noise only coarse control: in our experiments, the information it carries grows roughly with the logarithm of the number of competing fields, and one-step outputs remain rarely coherent across the model sizes tested. We propose CONDOR (Coupled-Noise Distillation for One-Step Readout). A noise-conditioned teacher is trained with a random number of masked positions and winner-take-all. A student proposes a one-step block, retains selected tokens, and learns from the block obtained when the teacher refills the other positions in several steps under the same noise field; a noise-free masked-LM term on the ground truth anchors the student. Human evaluation on TinyStories shows a large gain in one-step legality while different noise fields still yield different blocks, at one forward pass per block.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。