揭示掩码扩散模型的动态注意力机制及其性能优势
Empirical Analysis of Decoding Biases in Masked Diffusion Models
- 发现掩码扩散模型存在漂浮注意力现象，非固定聚焦
- 浅层用浮动标记构建全局结构，深层专注语义内容
- 解释其上下文学习强，知识任务性能翻倍于自回归模型
掩码扩散模型（MDMs）通过双向注意力与去噪过程，正缩小与自回归模型（ARMs）的性能差距。然而其内部注意力机制仍不清晰。本文研究发现，MDMs存在注意力漂浮现象：与注意力收敛至固定位置的ARMS不同，MDMs的注意力锚点在去噪步骤和网络层间动态分散。进一步分析表明，其具备浅层结构感知、深层内容聚焦的特性——浅层利用浮动标记构建全局结构框架，深层则更专注捕捉语义内容。实证显示，这种独特注意力模式为MDMs强大的上下文学习能力提供了机制解释，在知识密集型任务中性能较ARMS提升一倍。代码已开源：https://github.com/NEUIR/Uncode。
原文摘要 · Abstract (English)
Masked diffusion models (MDMs), which leverage bidirectional attention and a denoising process, are narrowing the performance gap with autoregressive models (ARMs). However, their internal attention mechanisms remain under-explored. This paper investigates the attention behaviors in MDMs, revealing the phenomenon of Attention Floating. Unlike ARMs, where attention converges to a fixed sink, MDMs exhibit dynamic, dispersed attention anchors that shift across denoising steps and layers. Further analysis reveals its Shallow Structure-Aware, Deep Content-Focused attention mechanism: shallow layers utilize floating tokens to build a global structural framework, while deeper layers allocate more capability toward capturing semantic content. Empirically, this distinctive attention pattern provides a mechanistic explanation for the strong in-context learning capabilities of MDMs, allowing them to double the performance compared to ARMs in knowledge-intensive tasks. All codes are available at https://github.com/NEUIR/Uncode.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。