揭示了掩码扩散模型中动态漂浮注意力机制，解释其强上下文学习能力。
Revealing the Attention Floating Mechanism in Masked Diffusion Models
- 发现掩码扩散模型存在动态漂浮注意力，随去噪步骤和层深变化
- 浅层用浮动标记构建全局结构，深层专注捕捉语义内容
- 使模型在知识密集任务中性能翻倍，适合研究生成模型机制者
掩码扩散模型（MDMs）通过双向注意力与去噪过程，正缩小与自回归模型（ARMs）的性能差距。然而其内部注意力机制仍不明确。本文研究发现MDMs存在注意力漂浮现象：与注意力趋于固定锚点的ARMs不同，MDMs的注意力锚点在去噪步骤与层间动态分散移动。进一步分析表明其具有浅层结构感知、深层内容聚焦的特性：浅层利用浮动标记建立全局结构框架，深层则更专注于捕捉语义内容。实证显示，这种独特注意力模式为MDMs强大的上下文学习能力提供了机制解释，使其在知识密集型任务中性能较ARMs提升一倍。所有代码与数据集见https://github.com/NEUIR/Attention-Floating。
原文摘要 · Abstract (English)
Masked diffusion models (MDMs), which leverage bidirectional attention and a denoising process, are narrowing the performance gap with autoregressive models (ARMs). However, their internal attention mechanisms remain under-explored. This paper investigates the attention behaviors in MDMs, revealing the phenomenon of Attention Floating. Unlike ARMs, where attention converges to a fixed sink, MDMs exhibit dynamic, dispersed attention anchors that shift across denoising steps and layers. Further analysis reveals its Shallow Structure-Aware, Deep Content-Focused attention mechanism: shallow layers utilize floating tokens to build a global structural framework, while deeper layers allocate more capability toward capturing semantic content. Empirically, this distinctive attention pattern provides a mechanistic explanation for the strong in-context learning capabilities of MDMs, allowing them to double the performance compared to ARMs in knowledge-intensive tasks. All codes and datasets are available at https://github.com/NEUIR/Attention-Floating.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。