让视觉Transformer的注意力头能互相交流，提升图像识别效果。
Beyond Isolated Heads: Multi-Overlapped-Head Self-Attention for Vision Transformers
- 用软重叠方式替代传统注意力头的硬分割，实现头间信息互通。
- 在CIFAR-10、ImageNet-1k等数据集上显著提升准确率，计算开销几乎不变。
- 适合追求模型性能优化的视觉Transformer研究者和开发者。
多头自注意力（MHSA）是视觉变换器的核心，通过将标记投影到独立子空间来捕捉多样化特征表示。然而，标准MHSA严格隔离各注意力头，导致注意力计算过程中无法进行信息交换。本文提出多重叠头自注意力（MOHSA），用软重叠的划分方式取代硬分割。通过允许查询、键和值与邻近头的部分内容重叠，MOHSA在注意力机制内部直接促进丰富的头间通信。我们系统研究了多种重叠维度调度策略，找到最优配置。在多个Transformer架构上的实验表明，MOHSA在CIFAR-10、CIFAR-100、Tiny-ImageNet和ImageNet-1k数据集上均优于标准MHSA，带来显著性能提升且计算开销可忽略。代码链接：https://github.com/ZTX-100/MOHSA。
原文摘要 · Abstract (English)
Multi-Head Self-Attention (MHSA) is the cornerstone of Vision Transformers, allowing models to capture diverse feature representations by projecting tokens into independent subspaces. However, the standard MHSA strictly isolates these heads, preventing any information exchange during the attention computation itself. In this paper, we propose Multi-Overlapped-Head Self-Attention (MOHSA), a novel mechanism that replaces the hard division of attention heads with a soft, overlapping division. By allowing queries, keys, and values to partially overlap with those of adjacent heads, MOHSA fosters rich inter-head communication directly within the attention mechanism. We extensively investigate various overlap dimension scheduling strategies to identify optimal configurations. Comprehensive experiments in several Transformer architectures demonstrate that MOHSA outperforms standard MHSA on the CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1k datasets, offering a significant performance boost with negligible computational overhead. Code link: https://github.com/ZTX-100/MOHSA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。