提出无需训练的并行解码方法，提升扩散语言模型生成效率
DAPD: Dependency-Aware Parallel Decoding via Attention for Diffusion LLMs
- 利用自注意力构建词元依赖图，识别强弱关联关系
- 通过独立集选择并行解码词元，实现更优准确率与步数平衡
- 无需额外模型或重训练，适合追求高效生成的研究者
扩散语言模型（dLLMs）的并行解码困难，因每步仅提供词元的边际分布，同时解码多个词元需考虑词元间依赖。本文提出无训练的依赖感知并行解码（DAPD），通过自注意力机制构建被掩码词元间的条件依赖图。每轮迭代中，图内边表示强交互，非边表示弱依赖。并行解码转化为在图上选取独立集并行解码选定词元。该方法避免同时更新强耦合词元，无需辅助模型或重训练。在LLaDA和Dream上的实验表明，DAPD在准确率-步数权衡上优于现有方法，支持更全局分布的并行更新，更好发挥dLLMs任意顺序生成能力。项目地址：https://ai-isl.github.io/dapd
原文摘要 · Abstract (English)
Parallel decoding for Diffusion LLMs (dLLMs) is difficult because each denoising step provides only token-wise marginal distributions, while unmasking multiple tokens simultaneously requires accounting for inter-token dependencies. We propose Dependency-Aware Parallel Decoding (DAPD), a simple, training-free decoding method that uses self-attention to induce a conditional dependency graph over masked tokens. At each iteration, edges in this graph capture strong token interactions, while non-edges indicate weak dependence. Parallel decoding is then reduced to selecting an independent set on the graph and unmasking the selected tokens in parallel. This avoids co-updating strongly coupled tokens without auxiliary models or retraining. Experiments on LLaDA and Dream show that DAPD improves the accuracy-steps trade-off over existing methods and enables more globally distributed parallel updates that better exploit the any-order generation capability of dLLMs. The project is available at https://ai-isl.github.io/dapd
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。