用一步生成取代多步迭代，让语言模型推理快500倍
DLM-One: Diffusion Language Models for One-Step Sequence Generation
- 通过分数蒸馏将学生模型输出对齐教师模型得分函数
- 在DiffuSeq上实现最高500倍推理速度提升
- 适合追求高效生成的工业级文本生成场景
本文提出DLM-One，一种基于分数蒸馏的连续扩散语言模型（DLM）一歩序列生成框架。该方法通过将学生模型输出在连续词嵌入空间中的分数与预训练教师模型的分数函数对齐，消除了迭代优化的必要性。我们在代表性的连续DLM——DiffuSeq上进行了全面实验，结果表明，DLM-One在保持与教师模型相当性能的前提下，推理时间最快可提升约500倍。我们还分析了该方法在多个数据集上的表现，初步揭示其泛化能力与实际应用潜力。研究结果表明，一步扩散生成是实现高效高质量语言生成的有前景方向，有助于推动基于嵌入空间的连续扩散模型在自然语言处理中的广泛应用。
原文摘要 · Abstract (English)
This paper introduces DLM-One, a score-distillation-based framework for one-step sequence generation with continuous diffusion language models (DLMs). DLM-One eliminates the need for iterative refinement by aligning the scores of a student model's outputs in the continuous token embedding space with the score function of a pretrained teacher DLM. We investigate whether DLM-One can achieve substantial gains in sampling efficiency for language modeling. Through comprehensive experiments on DiffuSeq -- a representative continuous DLM -- we show that DLM-One achieves up to ~500x speedup in inference time while maintaining competitive performance on benchmark text generation tasks used to evaluate the teacher models. We further analyze the method's empirical behavior across multiple datasets, providing initial insights into its generality and practical applicability. Our findings position one-step diffusion as a promising direction for efficient, high-quality language generation and broader adoption of continuous diffusion models operating in embedding space for natural language processing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。