提出分簇注意力引导的并行解码，让语言模型一次更新整段高置信度内容。
Cluster-Level Attention-Guided Parallel Decoding for Masked Diffusion Language Models
- 将高置信度预测聚成连续片段，以块为单位并行更新。
- 利用自注意力图识别片段间依赖，避免冲突，提升并行效率。
- 无需训练，在多个模型上实现1.77到8.47倍加速，适合推理加速场景。
掩码扩散语言模型（MDLMs）通过在每步去噪中同时预测所有被遮蔽位置实现并行解码，但现有无训练采样器通常以单个词符为粒度决定提交内容。本文重新审视这一粒度，观察到可靠预测常以连续高置信度片段形式出现，表明并行提交单位可大于单个词符。首先，将相邻高置信度候选合并为置信度驱动的簇（CICs），作为跨度级更新单元；随后，利用同一前向传播中的自注意力图估计簇间依赖关系，实现冲突感知的互兼容簇选择，用于并行提交。该方法形成无需训练的集群级解码器——CLAD，适用于MDLM。在LLaDA与Dream模型家族上，于四个推理与代码生成基准测试中，CLAD相较原始解码实现1.77倍至8.47倍加速，多数情况下任务准确率保持相当。
原文摘要 · Abstract (English)
Masked diffusion language models (MDLMs) enable parallel decoding by predicting all masked positions at each denoising step, yet existing training-free samplers usually decide which positions to commit at token-level granularity. We revisit this granularity and observe that reliable predictions often emerge as contiguous high-confidence spans, suggesting that the unit of parallel commitment can be larger than a single token. We first group adjacent high-confidence candidates into confidence-induced clusters (CICs) as span-level update units. We then use self-attention maps from the same forward pass to estimate inter-cluster dependencies, enabling conflict-aware selection of mutually compatible CICs for parallel commitment. This yields CLAD (Cluster-Level Attention-Guided Decoding), a training-free cluster-level decoder for MDLMs. Experiments on LLaDA and Dream model families across four reasoning and code-generation benchmarks show that CLAD achieves 1.77x--8.47x speedups over Vanilla decoding while maintaining broadly comparable task accuracy in most settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。