arXiv:2508.16884cs.CVcs.NE2025-08被引 8

轻量级模型融合卷积与多尺度自注意力,兼顾效率与性能。

A Lightweight Convolution and Vision Transformer integrated model with Multi-scale Self-attention Mechanism

  • 采用稀疏注意力与卷积块结合,降低计算开销。
  • 在ImageNet-1K上仅用0.8 GFLOPs达76.3%准确率。
  • 适合移动端或资源受限场景的视觉任务部署。

视觉变换器(ViT)凭借强大的长距离依赖建模能力在计算机视觉任务中广泛应用,但其庞大的模型尺寸和较弱的局部特征建模能力限制了实际应用。为平衡下游视觉任务中的计算效率与性能,本文提出一种高效的ViT模型——SAEViT,包含稀疏注意力机制和卷积模块。具体地,提出稀疏聚合注意力(SAA)模块,通过自适应稀疏采样并利用反卷积恢复特征图,显著降低注意力计算复杂度;设计通道交互式前馈网络(CIFFN),通过特征分解与重分配增强通道间信息交换,缓解传统前馈网络冗余;构建嵌入深度可分离卷积块(DWSConv)的分层金字塔结构,进一步强化卷积特征提取。大量实验表明,SAEViT在ImageNet-1K分类任务中分别以0.8 GFLOPs和1.3 GFLOPs实现76.3%和79.6%的Top-1准确率,证明其为基础视觉任务提供轻量化解决方案。

原文摘要 · Abstract (English)

Vision Transformer (ViT) has prevailed in computer vision tasks due to its strong long-range dependency modelling ability. \textcolor{blue}{However, its large model size and weak local feature modeling ability hinder its application in real scenarios. To balance computation efficiency and performance in downstream vision tasks, we propose an efficient ViT model with sparse attention (dubbed SAEViT) and convolution blocks. Specifically, a Sparsely Aggregated Attention (SAA) module has been proposed to perform adaptive sparse sampling and recover the feature map via deconvolution operation,} which significantly reduces the computational complexity of attention operations. In addition, a Channel-Interactive Feed-Forward Network (CIFFN) layer is developed to enhance inter-channel information exchange through feature decomposition and redistribution, which mitigates the redundancy in traditional feed-forward networks (FFN). Finally, a hierarchical pyramid structure with embedded depth-wise separable convolutional blocks (DWSConv) is devised to further strengthen convolutional features. Extensive experiments on mainstream datasets show that SAEViT achieves Top-1 accuracies of 76.3\% and 79.6\% on the ImageNet-1K classification task with only 0.8 GFLOPs and 1.3 GFLOPs, respectively, demonstrating a lightweight solution for fundamental vision tasks.

轻量模型视觉变换器多尺度注意力卷积融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。