SAG-ViT通过图注意力增强多尺度图像块表示,提升视觉Transformer的分类性能。
SAG-ViT: A Scale-Aware, High-Fidelity Patching Approach with Graph Attention for Vision Transformers
- 用图注意力构建多尺度特征块的拓扑关系,保留更丰富的语义信息。
- 在ImageNet等数据集上准确率优于标准ViT,提升约1.8个百分点。
- 适合需要高精度图像分类且关注多尺度特征建模的研究者。
视觉变换器(ViTs)通过自注意力机制捕捉图像块间的复杂模式与长程依赖,重新定义了图像分类任务。然而,其高效融合多尺度特征表示的能力仍受限,而卷积神经网络(CNNs)凭借层次结构天然具备此优势。图变换器虽在该方向取得进展,但常因冗余或不相关区域导致空间层次信息丢失。为此,本文提出SAG-ViT:一种具备尺度感知能力的图注意力视觉变换器,融合了CNN的多尺度特性、ViT的表征能力及图注意力块处理机制,实现更丰富的上下文表示。以EfficientNetV2为骨干网络提取多尺度特征图,并将其划分为图像块以保留更多语义信息;利用空间与特征相似性构建图结构,通过图注意力网络(GAT)优化节点嵌入;再经由Transformer编码器捕获长程依赖与复杂交互。我们在多个领域基准数据集上评估了SAG-ViT,验证其在图像分类任务中的有效性。代码与权重已开源于https://github.com/shravan-18/SAG-ViT。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have redefined image classification by leveraging self-attention to capture complex patterns and long-range dependencies between image patches. However, a key challenge for ViTs is efficiently incorporating multi-scale feature representations, which is inherent in convolutional neural networks (CNNs) through their hierarchical structure. Graph transformers have made strides in addressing this by leveraging graph-based modeling, but they often lose or insufficiently represent spatial hierarchies, especially since redundant or less relevant areas dilute the image's contextual representation. To bridge this gap, we propose SAG-ViT, a Scale-Aware Graph Attention ViT that integrates multi-scale feature capabilities of CNNs, representational power of ViTs, graph-attended patching to enable richer contextual representation. Using EfficientNetV2 as a backbone, the model extracts multi-scale feature maps, dividing them into patches to preserve richer semantic information compared to directly patching the input images. The patches are structured into a graph using spatial and feature similarities, where a Graph Attention Network (GAT) refines the node embeddings. This refined graph representation is then processed by a Transformer encoder, capturing long-range dependencies and complex interactions. We evaluate SAG-ViT on benchmark datasets across various domains, validating its effectiveness in advancing image classification tasks. Our code and weights are available at https://github.com/shravan-18/SAG-ViT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。