用语义驱动的代表令牌,让视觉Transformer更高效地捕捉全局信息。
Representative Attention For Vision Transformers

- 基于表示空间动态生成代表令牌,替代固定坐标分块
- 将计算复杂度从二次降低为线性,保持全局上下文建模能力
- 适用于图像分类、检测与分割,提升多种ViT模型性能
线性注意力已成为突破视觉Transformer中密集自注意力二次计算开销的有前景方向。现有方法通常将空间标记压缩为由预定义空间布局生成的中间代理标记,导致标记压缩仍受图像坐标束缚,而非视觉内容的语义结构。为此,我们提出代表注意力(RPAttention),一种线性全局注意力机制,直接在表示空间执行标记压缩。不同于基于固定空间划分构建代理标记,它通过轻量级的‘收集-交互-分发’范式,动态形成一组可学习的代表性标记,使语义相关的区域即使空间上相距遥远也能通信。空间标记首先通过基于相似性的竞争路由软性汇聚到代表标记;代表标记在紧凑的潜在空间内进行全局交互,再通过查询驱动的交叉注意力将优化信息广播回所有空间标记。通过将坐标驱动的聚合替换为表示驱动的压缩,RPAttention在保持全局感受野的同时,自适应对齐标记通信与输入内容结构。该方法将主导的标记交互复杂度从二次降低为线性,同时维持强大的全局上下文建模能力。在多种视觉变压器骨干网络上的图像分类、目标检测和语义分割任务中,实验充分验证了该设计的有效性。
原文摘要 · Abstract (English)
Linear attention has emerged as a promising direction for scaling Vision Transformers beyond the quadratic cost of dense self-attention. A prevalent strategy is to compress spatial tokens into a compact set of intermediate proxies that mediate global information exchange. However, existing methods typically derive these proxy tokens from predefined spatial layouts, causing token compression to remain anchored to image coordinates rather than the semantic organization of visual content. To overcome this limitation, we propose Representative Attention (RPAttention), a linear global attention mechanism that performs token compression directly in representation space. Instead of constructing intermediate tokens from fixed spatial partitions, it dynamically forms a compact set of learned representative tokens to enable semantically related regions to communicate regardless of their spatial distance, by following a lightweight Gather-Interact-Distribute paradigm. Spatial tokens are first softly gathered into representative tokens through competitive similarity-based routing. The representatives then perform global interaction within a compact latent space, before broadcasting the refined information back to all spatial tokens via query-driven cross-attention. Via replacing coordinate-driven aggregation with representation-driven compression, RPAttention preserves global receptive fields while adaptively aligning token communication with the content structure of each input.RPAttention reduces the dominant token interaction complexity from quadratic to linear scaling with respect to the number of spatial tokens, while maintaining expressive global context modeling. Extensive experiments across diverse vision transformer backbones on image classification, object detection, and semantic segmentation demonstrate the effectiveness of our design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。