让语言模型自适应生成长度，提升效率与兼容性
Sequential Diffusion Language Models
- 统一词元与块预测，动态决定每步生成长度
- 仅用350万样本训练，吞吐比Qwen-2.5高2.1倍
- 可低成本改造现有模型，适合追求高效推理的场景
扩散语言模型（DLM）理论效率高，但受限于固定长度解码和与键值缓存（KV cache）不兼容。块扩散虽缓解问题，仍需固定块大小且训练成本高。本文提出下一序列预测（NSP），统一下一词元与下一区块预测，使模型在每步可自适应确定生成长度；当长度为1时，退化为标准词元预测。基于NSP，我们提出顺序扩散语言模型（SDLM），可低成本微调预训练自回归语言模型（ALM）。SDLM在固定大小掩码块内进行扩散推理，但根据模型置信度动态解码连续子序列，保持与KV缓存兼容，并增强对序列中不确定性和语义变化的鲁棒性。实验表明，SDLM仅用350万训练样本即可达到或超越强自回归基线，吞吐量比Qwen-2.5高2.1倍。值得注意的是，SDLM-32B模型展现出更显著的效率提升，验证了该建模范式的强可扩展性。
原文摘要 · Abstract (English)
Diffusion language models (DLMs) have strong theoretical efficiency but are limited by fixed-length decoding and incompatibility with key-value (KV) caches. Block diffusion mitigates these issues, yet still enforces a fixed block size and requires expensive training. We introduce Next Sequence Prediction (NSP), which unifies next-token and next-block prediction, enabling the model to adaptively determine the generation length at each step. When the length is fixed to 1, NSP reduces to standard next-token prediction. Building on NSP, we propose Sequential Diffusion Language Model (SDLM), which can retrofit pre-trained autoregressive language models (ALMs) at minimal cost. Specifically, SDLM performs diffusion inference within fixed-size mask blocks, but dynamically decodes consecutive subsequences based on model confidence, thereby preserving KV-cache compatibility and improving robustness to varying uncertainty and semantics across the sequence. Experiments show that SDLM matches or surpasses strong autoregressive baselines using only 3.5M training samples, while achieving 2.1 higher throughput than Qwen-2.5. Notably, the SDLM-32B model delivers even more pronounced efficiency gains, demonstrating the strong scalability potential of our modeling paradigm. Project page and codes: https://github.com/OpenGVLab/SDLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。