融合图卷积与分层视觉变压器,提升图像分类精度。
Hierarchical Vision Transformer Enhanced by Graph Convolutional Network for Image Classification
- 分层设计捕捉局部与全局特征,结合小块与大块图像信息。
- 用图卷积提取局部结构,替代传统1D位置编码,增强空间感知。
- 在3个真实数据集上超越现有方法,适合追求高精度图像分类的场景。
视觉变换器(ViT)通过自注意力机制推动了图像分类的新突破,而图卷积网络(GCN)在数据表示与分析中也取得成功。然而,两者仍面临关键挑战:(1)ViT的补丁尺寸对预测准确性影响显著,如何合理选择或融合不同尺寸的补丁尚不明确;(2)1D位置嵌入难以准确捕捉补丁的空间结构信息;(3)GCN擅长建模局部连接关系,但缺乏对全局图结构的建模能力;反之,ViT可建模全局关系,却无法有效表达局部结构。为此,本文提出基于图卷积增强的分层视觉变换器(GCN-HViT)。该模型在每一层级内建模补丁间的全局交互,并跨层级建模小补丁与大补丁的层次关系。同时,引入图卷积作为局部特征提取器,为每个补丁生成2D空间位置嵌入,实现局部信息交互。大量实验在3个真实数据集上验证了其性能达到当前最优水平。
原文摘要 · Abstract (English)
Vision Transformer (ViT) has brought new breakthroughs to the field of image classification by introducing the self-attention mechanism and Graph Convolutional Networks(GCN) have been proposed and successfully applied in data representation and analysis. However, there are key challenges which limit their further development: (1) The patch size selected by ViT is crucial for accurate predictions, which raises a natural question: How to select the size of patches properly or how to comprehensively combine small patches and larger patches; (2) While the spatial structure information is important in vision tasks, the 1D position embeddings fails to capture the spatial structure information of patches more accurately; (3) The GCN can capture the local connectivity relationships between image nodes, but it lacks the ability to capture global graph structural information. On the contrary, the self-attention mechanism of ViT can draw the global relation on image patches, but it is unable to model the local structure of image. To overcome such limitations, we propose the Hierarchical Vision Transformer Enhanced by Graph Convolutional Network (GCN-HViT) for image classification. Specifically, the Hierarchical ViT we designed can model patch-wise information interactions on a global scale within each level and model hierarchical relationships between small patches and large patches across multiple levels. In addition, the proposed GCN method functions as a local feature extractor to obtain the local representation of each image patch which serves as a 2D position embedding of each patch in the 2D space. Meanwhile, it models patch-wise information interactions on a local scale within each level. Extensive experiments on 3 real-world datasets demonstrate that GCN-HViT achieves state-of-the-art performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。