提出依赖导向采样器，提升掩码扩散语言模型生成质量
DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models
- 基于注意力矩阵捕捉词间依赖关系，指导掩码位置更新
- 在代码生成与数学推理任务上表现优于现有方法
- 无需训练即可融合现有并行采样策略，效率质量兼备
掩码扩散语言模型（MDLMs）作为新兴的语言建模范式，具备灵活的生成动态和高效的并行解码能力。然而，现有的预训练MDLM解码策略主要依赖于词级不确定性指标，忽视了序列级信息和词间依赖关系。为此，本文提出无训练的依赖导向采样器（DOS），利用Transformer层中的注意力矩阵近似词间依赖，更新掩码位置时侧重未掩码词的信息。实验表明，DOS在代码生成与数学推理任务上持续取得更优性能。此外，DOS可无缝集成至现有并行采样方法，提升生成效率而不损失质量。
原文摘要 · Abstract (English)
Masked diffusion language models (MDLMs) have recently emerged as a new paradigm in language modeling, offering flexible generation dynamics and enabling efficient parallel decoding. However, existing decoding strategies for pre-trained MDLMs predominantly rely on token-level uncertainty criteria, while largely overlooking sequence-level information and inter-token dependencies. To address this limitation, we propose Dependency-Oriented Sampler (DOS), a training-free decoding strategy that leverages inter-token dependencies to inform token updates during generation. Specifically, DOS exploits attention matrices from transformer blocks to approximate inter-token dependencies, emphasizing information from unmasked tokens when updating masked positions. Empirical results demonstrate that DOS consistently achieves superior performance on both code generation and mathematical reasoning tasks. Moreover, DOS can be seamlessly integrated with existing parallel sampling methods, leading to improved generation efficiency without sacrificing generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。