压缩扩散模型中的冗余掩码令牌，提升生成速度与长文本处理能力
Elastic-dLLM: Position Preserving Context Compression and Augmentation of Diffusion LLMs
- 通过保留位置信息压缩冗余掩码令牌，减少重复计算
- 在有限输入长度下实现类似上下文折叠的长序列扩展，加速解码
- 为块状扩散模型添加保护终端掩码，提升生成质量且开销极小
与逐个生成词元的自回归模型不同，扩散语言模型（dLLMs）会联合去噪一批[掩码]词元，并在每一步采样一个或多个词元；尽管支持并行解码，但大尺寸掩码块带来了显著计算开销。我们观察到，大量计算耗在重复处理先前上下文和具有相同特征表示的多个[掩码]词元上，表明存在巨大计算冗余。本文从[掩码]词元视角重新审视这一问题，通过系统分析验证了其冗余性，同时揭示了其在提供结构信息中的关键作用。基于此，提出位置保持的[掩码]词元压缩与终端感知增强方法。该方法通过压缩冗余[掩码]计算，加速解码，并在输入长度受限条件下，自然扩展至全序列扩散模型（如LLaDA-8B-Instruct和LLaDA-1.5）的长上下文处理。此外，对块状扩散模型（如LLaDA2.0-mini），通过引入保护终端[掩码]词元，在几乎无额外开销下提升生成质量。
原文摘要 · Abstract (English)
Unlike autoregressive models, which generate one token at a time, dLLMs denoise a chunk of [MASK] tokens jointly and sample one or more tokens per step; despite enabling parallel decoding, this process incurs substantial computational cost due to the large chunk size of masked tokens. We observe that much of this cost is spent on repeatedly processing the preceding context and many [MASK] tokens with the same feature representations, indicating considerable computational redundancy. In this work, we revisit dLLM's redundancy from the perspective of [MASK] tokens. Through systematic analysis, we verify the redundancy of [MASK] tokens while revealing their critical role in providing structural information. Guided by these findings, we propose position-preserving [MASK] token compression and terminal-aware augmentation. By compressing redundant [MASK] computation, this approach accelerates decoding and further provides a natural extension toward context-folding-like long-context scaling under limited input-length constraints for full-sequence dLLMs such as LLaDA-8B-Instruct and LLaDA-1.5. Moreover, for block dLLMs such as LLaDA2.0-mini, it augments the context with a protected terminal [MASK] token to enhance generation quality with negligible overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。