arXiv:2503.21262cs.CV2025-03

用轻量级状态空间模块替代传统卷积,高效建模视觉长距离依赖。

vGamba: Attentive State Space Bottleneck for efficient Long-range Dependencies in Visual Recognition

  • 设计Gamba细胞,融合2D位置感知与注意力空间上下文,单块实现长程建模。
  • 在2048×2048分辨率下比BotNet快2.07倍,显存降低93.8%至1.03GB。
  • 适合高分辨率视觉任务,兼具效率与可扩展性,优于现有SNN和Transformer方案。

高效捕捉长距离依赖(LRD)是视觉识别的核心挑战,状态空间模型(SSMs)近年来成为替代自注意力的有前景方案。然而,将SSMs融入基于CNN的瓶颈结构仍具挑战,现有方法需复杂预处理和多个SSM副本,实用性受限。本文提出vGamba,一种混合视觉主干网络,以单一轻量级SSM块——Gamba细胞,取代标准瓶颈卷积,该细胞结合2D位置感知与注意力空间上下文(ASC)模块,实现高效LRD建模。在多种下游视觉任务中,vGamba性能媲美VMamba、ViM等基于SSM的模型,同时计算与内存效率显著优于Bottleneck Transformer(BotNet)。例如,在2048×2048分辨率下,vGamba比BotNet快2.07倍,峰值GPU显存从16.78GB降至1.03GB(减少93.8%),且随分辨率增长近似线性扩展,与ResNet-50相当。结果表明,Gamba瓶颈有效克服了BotNet全局建模的内存与计算瓶颈,为高分辨率视觉任务提供实用且可扩展的骨干网络。

原文摘要 · Abstract (English)

Capturing long-range dependencies (LRD) efficiently is a core challenge in visual recognition, and state-space models (SSMs) have recently emerged as a promising alternative to self-attention for addressing it. However, adapting SSMs into CNN-based bottlenecks remains challenging, as existing approaches require complex pre-processing and multiple SSM replicas per block, limiting their practicality. We propose vGamba, a hybrid vision backbone that replaces the standard bottleneck convolution with a single lightweight SSM block, the Gamba cell, which incorporates 2D positional awareness and an attentive spatial context (ASC) module for efficient LRD modeling. Results on diverse downstream vision tasks demonstrate competitive accuracy against SSM-based models such as VMamba and ViM, while achieving significantly improved computation and memory efficiency over Bottleneck Transformer (BotNet). For example, at $2048 \times 2048$ resolution, vGamba is $2.07 \times$ faster than BotNet and reduces peak GPU memory by 93.8% (1.03GB vs. 16.78GB), scaling near-linearly with resolution comparable to ResNet-50. These results demonstrate that Gamba Bottleneck effectively overcomes the memory and compute constraints of BotNet global modeling, establishing it as a practical and scalable backbone for high-resolution vision tasks.

视觉识别状态空间长程依赖轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。