提出SEMA，解决注意力分散问题，实现高效且聚焦的视觉注意力。
SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging
- 通过令牌定位避免注意力分散，结合平均聚合捕捉全局特征。
- 在ImageNet-1K上超越现有视觉Mamba模型，在更大图像下仍保持高效。
- 适合追求高效、可扩展视觉注意力的工程师与研究者。
注意力是Transformer的核心组件。然而，传统全注意力机制存在输入规模的二次复杂度，而线性注意力又难以实现聚焦。本文给出广义注意力的数学定义，将标准Softmax注意力与线性注意力统一于同一框架。证明广义注意力具有分散性：当键的数量趋于无穷时，查询对所有键赋予相等权重。基于此分散特性及Mamba类注意力的最新进展，设计了可扩展高效的类Mamba注意力（SEMA），利用令牌定位防止分散并保持聚焦，辅以理论一致的算术平均捕捉全局信息。在ImageNet-1K上的分类实验表明，SEMA在模型参数量相近的情况下，相较于近期视觉Mamba模型，在更大图像尺度下仍表现更优，是一种有效的替代方案。
原文摘要 · Abstract (English)
Attention is the critical component of a transformer. Yet the quadratic computational complexity of vanilla full attention in the input size and the inability of its linear attention variant to focus have been challenges for computer vision tasks. We provide a mathematical definition of generalized attention and formulate both vanilla softmax attention and linear attention within the general framework. We prove that generalized attention disperses, that is, as the number of keys tends to infinity, the query assigns equal weights to all keys. Motivated by the dispersion property and recent development of Mamba form of attention, we design Scalable and Efficient Mamba like Attention (SEMA) which utilizes token localization to avoid dispersion and maintain focusing, complemented by theoretically consistent arithmetic averaging to capture global aspect of attention. We support our approach on Imagenet-1k where classification results show that SEMA is a scalable and effective alternative beyond linear attention, outperforming recent vision Mamba models on increasingly larger scales of images at similar model parameter sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。