将自回归模型适配为扩散语言模型,实现高效长文本生成。
From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
- 从逐词生成逐步过渡到分块扩散,保持上下文因果性。
- 70亿参数模型在长文本推理上达到同类最佳表现。
- 适合需要快速构建高性能生成模型的研究者使用。
扩散语言模型(DLM)支持快速生成,但从头训练大模型成本高昂。作为实用捷径,将现成的自回归(AR)模型权重适配为DLM可快速赋予其强长文本生成能力。现有方法或修改输出概率,或随机扩展注意力掩码至全序列,或直接移植权重至分块扩散结构,均未解决适配终点与优化路径问题。为此,我们基于分块扩散范式,提出从块大小1逐步过渡到最终状态的系统化适配路径:保留前缀的因果注意力、维持自回归引导的并行适配机制,并渐进增大生成块尺寸以实现平滑过渡。在此基础上,适配性能在多规模模型上均具竞争力。由此提出的NBDiff-7B继承了优秀长程建模与推理能力,在7B级DLM中达到当前最优表现。代码已公开:https://github.com/YuchuanTian/NBDiff。
原文摘要 · Abstract (English)
Diffusion Language Models (DLMs) enable fast generation, yet training large DLMs from scratch is costly. As a practical shortcut, adapting off-the-shelf Auto-Regressive (AR) model weights into a DLM could quickly equip the DLM with strong long-context generation capabilies. Prior "adaptation" attempts either modify logits or randomly grow attention masks to Full-Sequence diffusion, or simply transplant AR weights into a Block-Diffusion recipe, leaving two key questions unaddressed: where is the final destination of adaptation, and how to adapt better? For manifold benefits, we reframe the whole AR-to-DLM adaptation under the Block-Diffusion paradigm, transitioning from block size 1 to the final Block-Diffusion state. Concretely, the principled pathway of adaptation is designed as follows: we keep a context-causal path where causal attention is kept in the prefix, an efficient parallel adaptation procedure where an AR guidance is maintained, and gradual increment of the generation block size for a smoother transition. Built on these components, the adaptation is proved competitive on various models at different scales. With better adaptation, we propose NBDiff-7B that could inherit the long-context modeling and reasoning capabilities, and achieve state-of-the-art performance among the 7B-class DLMs. Codes: https://github.com/YuchuanTian/NBDiff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。