提出V2M模型,直接在二维空间处理图像,兼顾局部结构与计算效率。
V2M: Visual 2-Dimensional Mamba for Image Representation Learning
- 将状态空间模型推广到二维,同时考虑行和列的邻近信息
- 在ImageNet、COCO、ADE20K上均超越现有视觉骨干网络
- 适合追求高效且保留图像局部结构的视觉任务应用
Mamba因其灵活设计和高效的硬件表现,被广泛用于处理1维序列。近期研究尝试将Mamba应用于视觉领域,通过将2维图像分块并展平为1维序列。为弥补原始图像2维结构信息(如局部相似性)的损失,多数方法聚焦于设计不同顺序来逐个处理标记,仅部分缓解该问题。本文提出一种完整的解决方案——视觉二维Mamba(V2M),直接在2维空间中处理图像标记。我们首先将状态空间模型(SSM)推广至二维空间,生成下一状态时同时考虑两个方向上的相邻状态(如行列)。基于此二维SSM形式,构建V2M,并结合Mamba实现硬件高效的并行处理。所提V2M有效融合2维局部先验,同时继承Mamba的高效性与输入相关可扩展性。在ImageNet分类及下游视觉任务(包括COCO目标检测与实例分割、ADE20K语义分割)上的大量实验表明,其性能优于其他视觉骨干网络。
原文摘要 · Abstract (English)
Mamba has garnered widespread attention due to its flexible design and efficient hardware performance to process 1D sequences based on the state space model (SSM). Recent studies have attempted to apply Mamba to the visual domain by flattening 2D images into patches and then regarding them as a 1D sequence. To compensate for the 2D structure information loss (e.g., local similarity) of the original image, most existing methods focus on designing different orders to sequentially process the tokens, which could only alleviate this issue to some extent. In this paper, we propose a Visual 2-Dimensional Mamba (V2M) model as a complete solution, which directly processes image tokens in the 2D space. We first generalize SSM to the 2-dimensional space which generates the next state considering two adjacent states on both dimensions (e.g., columns and rows). We then construct our V2M based on the 2-dimensional SSM formulation and incorporate Mamba to achieve hardware-efficient parallel processing. The proposed V2M effectively incorporates the 2D locality prior yet inherits the efficiency and input-dependent scalability of Mamba. Extensive experimental results on ImageNet classification and downstream visual tasks including object detection and instance segmentation on COCO and semantic segmentation on ADE20K demonstrate the effectiveness of our V2M compared with other visual backbones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。