揭示Transformer自注意力的内在结构：对称性、方向性与涌现动态
The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
- 通过数学框架分析自注意力权重更新机制
- 双向训练产生对称矩阵，自回归训练导致方向性与列主导
- 适用于各类模型与模态，助力提升编码器性能
自注意力是Transformer架构的核心，但其权重矩阵中信息的嵌入方式及不同目标函数的影响仍不清晰。本文提出数学框架，推导自注意力权重更新的结构规律。研究表明，双向训练使权重矩阵呈现对称性，而自回归训练则引发方向性与列主导现象。理论结果在ModernBERT、GPT、LLaMA3、Mistral等多个模型及文本、视觉、音频多种模态上得到验证。进一步应用发现，对称初始化可提升编码器模型在语言任务上的表现。该分析为理解自注意力中的信息嵌入提供了新的理论视角，增强了Transformer模型的可解释性。
原文摘要 · Abstract (English)
Self-attention is essential to Transformer architectures, yet how information is embedded in the self-attention matrices and how different objective functions impact this process remains unclear. We present a mathematical framework to analyze self-attention matrices by deriving the structures governing their weight updates. Using this framework, we demonstrate that bidirectional training induces symmetry in the weight matrices, while autoregressive training results in directionality and column dominance. Our theoretical findings are validated across multiple Transformer models - including ModernBERT, GPT, LLaMA3, and Mistral - and input modalities like text, vision, and audio. Finally, we apply these insights by showing that symmetric initialization improves the performance of encoder-only models on language tasks. This mathematical analysis offers a novel theoretical perspective on how information is embedded through self-attention, thereby improving the interpretability of Transformer models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。