通过熵变化自适应划分块，提升扩散语言模型推理效率与质量。
Swordsman: Entropy-Driven Adaptive Block Partition for Efficient Diffusion Language Models
- 基于相邻词元间熵的变化动态识别语义边界，自适应分块。
- 在多个数据集上实现比基线更快的生成速度和更高的准确率。
- 无需训练，适配现有模型，适合追求高效推理的研究者。
块级解码通过块间序列去噪与块内并行去掩码，有效提升了扩散语言模型（DLMs）的推理速度与质量。然而，现有方法通常采用固定方式划分块，不可避免地割裂完整的语义或句法成分，导致性能下降。受熵减少假说（ERH）启发，我们发现成分边界更有利于不确定性降低，因此提出基于熵分析识别成分边界的Swordsman框架。Swordsman通过检测相邻词元间的熵变，自适应划分块，更好地对齐语义或句法成分边界。此外，它根据块内实时去掩码状态动态调整去掩码阈值，进一步提升效率与稳定性。作为无训练框架，Swordsman结合KV Cache，在多项评估中达到当前最优表现。
原文摘要 · Abstract (English)
Block-wise decoding effectively improves the inference speed and quality in diffusion language models (DLMs) by combining inter-block sequential denoising and intra-block parallel unmasking. However, existing block-wise decoding methods typically partition blocks in a rigid and fixed manner, which inevitably fragments complete semantic or syntactic constituents, leading to suboptimal performance. Inspired by the entropy reduction hypothesis (ERH), we recognize that constituent boundaries offer greater opportunities for uncertainty reduction, which motivates us to employ entropy analysis for identifying constituent boundaries. Therefore, we propose Swordsman, an entropy-driven adaptive block-wise decoding framework for DLMs. Swordsman adaptively partitions blocks by identifying entropy shifts between adjacent tokens to better align with semantic or syntactic constituent boundaries. In addition, Swordsman dynamically adjusts unmasking thresholds conditioned on the real-time unmasking status within a block, further improving both efficiency and stability. As a training-free framework, supported by KV Cache, Swordsman demonstrates state-of-the-art performance across extensive evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。