arXiv:2411.17473cs.CV2024-11ICCV被引 41

通过解耦频域特征,让小型视觉Mamba模型更高效地捕捉全局信息。

TinyViM: Frequency Decoupling for Tiny Hybrid Vision Mamba

  • 设计拉普拉斯混合器,只将低频特征输入Mamba模块
  • 在不同层级渐进减少高频分支维度,平衡高低频信息
  • 比同类模型快2-3倍,性能超越卷积与Transformer

Mamba在计算机视觉中展现出巨大潜力,因其对输入长度具有线性复杂度,可建模全局上下文。然而,现有轻量级基于Mamba的主干网络难以达到卷积或Transformer方法的性能。我们通过全面的谱分析和定量研究发现,在卷积-Mamba混合架构下,Mamba模块主要建模低频信息。基于此,我们提出一种新型拉普拉斯混合器,解耦特征频域成分,仅将低频分量输入Mamba模块。同时,考虑到特征冗余及不同阶段对高频细节与低频全局信息的不同需求,引入频率阶梯式结构(frequency ramp inception),逐步降低高频分支输入维度,实现各层高低频成分的高效权衡。结合轻量级卷积与高效拉普拉斯混合器,构建了一系列微型混合视觉Mamba模型TinyViM。TinyViM在图像分类、语义分割、目标检测和实例分割等多个下游任务中表现优异,显著优于同规模的卷积、Transformer和Mamba模型,且吞吐量为其他Mamba模型的2-3倍。代码已开源。

原文摘要 · Abstract (English)

Mamba has shown great potential for computer vision due to its linear complexity in modeling the global context with respect to the input length. However, existing lightweight Mamba-based backbones cannot demonstrate performance that matches Convolution or Transformer-based methods. By observing, we find that simply modifying the scanning path in the image domain is not conducive to fully exploiting the potential of vision Mamba. In this paper, we first perform comprehensive spectral and quantitative analyses, and verify that the Mamba block mainly models low-frequency information under Convolution-Mamba hybrid architecture. Based on the analyses, we introduce a novel Laplace mixer to decouple the features in terms of frequency and input only the low-frequency components into the Mamba block. In addition, considering the redundancy of the features and the different requirements for high-frequency details and low-frequency global information at different stages, we introduce a frequency ramp inception, i.e., gradually reduce the input dimensions of the high-frequency branches, so as to efficiently trade-off the high-frequency and low-frequency components at different layers. By integrating mobile-friendly convolution and efficient Laplace mixer, we build a series of tiny hybrid vision Mamba called TinyViM. The proposed TinyViM achieves impressive performance on several downstream tasks including image classification, semantic segmentation, object detection and instance segmentation. In particular, TinyViM outperforms Convolution, Transformer and Mamba-based models with similar scales, and the throughput is about 2-3 times higher than that of other Mamba-based models. Code is available at https://github.com/xwmaxwma/TinyViM.

视觉Mamba频域解耦轻量化高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。