将注意力机制置于弯曲空间中，用几何方法重新定义注意力。
RiemannFormer: A Framework for Attention in Curved Spaces
- 用度量张量、切空间等几何结构重构注意力机制
- 在视觉与语言模型上均显著优于基线
- 适合关注模型可解释性与几何深度学习的研究者
本研究旨在探索基于Transformer架构的进一步潜力，核心动机是为注意力机制提供几何解释。在该框架中，注意力主要涉及度量张量、切空间、内积及其相互关系，这些离散位置的量通过切向量的平行移动紧密关联。为提升学习效率，通过巧妙预设配置减少参数数量。此外，引入显式机制以抑制远距离值、强化局部邻域，弥补Transformer固有的局部归纳偏置缺失。实验结果表明，所提模块相较于基线表现显著提升。后续将在视觉与大语言模型上开展更多评估实验。
原文摘要 · Abstract (English)
This research endeavors to offer insights into unlocking the further potential of transformer-based architectures. One of the primary motivations is to offer a geometric interpretation for the attention mechanism in transformers. In our framework, the attention mainly involves metric tensors, tangent spaces, inner product, and how they relate to each other. These quantities and structures at discrete positions are intricately interconnected via the parallel transport of tangent vectors. To make the learning process more efficient, we reduce the number of parameters through ingenious predefined configurations. Moreover, we introduce an explicit mechanism to highlight a neighborhood by attenuating the remote values, given that transformers inherently neglect local inductive bias. Experimental results demonstrate that our modules deliver significant performance improvements relative to the baseline. More evaluation experiments on visual and large language models will be launched successively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。