用固定键替代动态键，让视觉变压器更高效
Static Key Attention in Vision
- 用固定键替代动态参数化键，简化注意力机制
- 在多个视觉任务上性能不降反升，部分超越标准自注意力
- 适合构建分层混合架构，平衡卷积与自注意力优势
视觉变换器的成功广泛归因于其动态参数化多头自注意力机制的表达能力。本文研究了在标准注意力机制中，用静态键替代动态参数化键的影响。结果表明，静态键注意力机制可达到甚至超过标准自注意力的性能。将静态键注意力模块融入Metaformer主干网络后发现，它在分层混合架构中作为中间阶段表现更优，能有效融合深度卷积与自注意力的优势。在多个视觉任务上的实验验证了静态键机制的有效性，表明注意力中的典型两步动态参数化可简化为单步，且在某些条件下不影响性能。
原文摘要 · Abstract (English)
The success of vision transformers is widely attributed to the expressive power of their dynamically parameterized multi-head self-attention mechanism. We examine the impact of substituting the dynamic parameterized key with a static key within the standard attention mechanism in Vision Transformers. Our findings reveal that static key attention mechanisms can match or even exceed the performance of standard self-attention. Integrating static key attention modules into a Metaformer backbone, we find that it serves as a better intermediate stage in hierarchical hybrid architectures, balancing the strengths of depth-wise convolution and self-attention. Experiments on several vision tasks underscore the effectiveness of the static key mechanism, indicating that the typical two-step dynamic parameterization in attention can be streamlined to a single step without impacting performance under certain circumstances.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。