用变分法分析注意力机制的球面流形结构，揭示其数学本质。
Inexact calculus of variations on the hyperspherical tangent bundle with connections to the attention mechanism
- 基于投影和ε扰动的近似变分法，研究单位超球面切空间上的优化
- 发现Transformer中每个标记的注意力可视为球面上的流映射
- 为理解注意力机制提供新数学视角，适合对几何与模型原理感兴趣的读者
我们通过在单位超球面及其切空间上进行拉格朗日优化，提供了一套理论数学基础。由于方法基于投影且对泛函优化引入ε型扰动，因此属于非精确方法。本文将该框架与注意力机制及Transformer联系起来，因为每个标记在高维单位球面上的注意力可被看作其切纤维中的流映射。本工作的动机主要有两点：一是研究注意力机制在其流映射下的性质及其与传统变分法和拉格朗日优化的关系；二是探索一系列适用于单位超球面的变分法，以更广阔的数学视角逼近变分问题。
原文摘要 · Abstract (English)
We offer a theoretical mathematical background through Lagrangian optimization on the unit hyperspherical manifold and its tangential structure. Our methods can be categorized as inexact since our methods are projection-based and since we will perturb the functional optimization with epsilon-type quantities. We draw connections to the attention mechanism and the Transformer since it exists as a flow map in the tangent fiber for each token along the high-dimensional unit sphere. Our motivation for this work is primarily twofold: we study the attention mechanism under its flow map and its relations to traditional calculus of variations and Lagrangian optimization; and we study a range of calculus of variations on the unit hypersphere that appeal to a broader mathematical lens in approximating, variational contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。