通过预测词元依赖关系，实现更高质量的并行文本生成
Dependency-Guided Parallel Decoding in Discrete Diffusion Language Models
- 用轻量级模型预测词元间依赖关系，指导并行解码
- 在Dream-7B上实现1.7~2.2倍加速，准确率不降反升
- 适合追求高效生成且关注输出质量的研究者
离散扩散语言模型（dLLMs）通过并行解码多个词元加速文本生成。然而，这种并行方式引入分布偏差：它用单个词元边缘分布的乘积近似联合条件分布，当被选中的词元存在强依赖时会降低输出质量。本文提出DEMASK（DEpendency-guided unMASKing），一个附加于dLLM最终隐藏状态的轻量级依赖预测器。仅需一次前向传播，即可估计被掩码位置间的成对条件影响。基于预测结果，贪婪算法选出累积依赖度受控的位置进行同时解码。在次可加性假设下，我们证明该方法可控制其并行采样与模型联合分布之间的总变差距离。实验表明，DEMASK在Dream-7B上实现1.7~2.2倍加速，同时保持或优于基于置信度和KL的基线模型的准确率。
原文摘要 · Abstract (English)
Discrete diffusion language models (dLLMs) accelerate text generation by unmasking multiple tokens in parallel. However, parallel decoding introduces a distributional mismatch: it approximates the joint conditional using a fully factorized product of per-token marginals, which degrades output quality when selected tokens are strongly dependent. We propose DEMASK (DEpendency-guided unMASKing), a lightweight dependency predictor that attaches to the final hidden states of a dLLM. In a single forward pass, it estimates pairwise conditional influences between masked positions. Using these predictions, a greedy selection algorithm identifies positions with bounded cumulative dependency for simultaneous unmasking. Under a sub-additivity assumption, we prove this bounds the total variation distance between our parallel sampling and the model's joint. Empirically, DEMASK achieves 1.7-2.2$\times$ speedup on Dream-7B while matching or improving accuracy compared to confidence-based and KL-based baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。