揭示自注意力的本质是分布投影，统一解释了Transformer的结构设计。
Self-Attention as Distributional Projection: A Unified Interpretation of Transformer Architecture
- 将词共现统计投影到序列上下文中，推导出自注意力机制
- 查询-键-值结构自然产生于方向性关系建模需求
- 位置编码与多头注意力是投影原则的有序扩展，非随意设计
本文通过连接分布语义原理，提出了自注意力机制的数学解释。我们证明，自注意力源于将语料库级别的共现统计量投影到序列上下文中。从GloVe嵌入所依赖的共现矩阵出发，展示了该投影如何自然捕捉上下文影响；查询-键-值机制作为不对称扩展，用于建模方向性关系。位置编码与多头注意力则作为这一投影原则的结构化改进。分析表明，Transformer架构的特定代数形式并非任意设计，而是由这些投影原则所决定。
原文摘要 · Abstract (English)
This paper presents a mathematical interpretation of self-attention by connecting it to distributional semantics principles. We show that self-attention emerges from projecting corpus-level co-occurrence statistics into sequence context. Starting from the co-occurrence matrix underlying GloVe embeddings, we demonstrate how the projection naturally captures contextual influence, with the query-key-value mechanism arising as the natural asymmetric extension for modeling directional relationships. Positional encodings and multi-head attention then follow as structured refinements of this same projection principle. Our analysis demonstrates that the Transformer architecture's particular algebraic form follows from these projection principles rather than being an arbitrary design choice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。