用Mamba架构实现高效全景分割,兼顾长程建模与多尺度特征。
MambaPanoptic: A Vision Mamba-based Structured State Space Framework for Panoptic Segmentation

- 基于Mamba块构建线性复杂度的特征金字塔,支持全局上下文建模。
- 在Cityscapes和COCO上达到优于基线模型的性能,参数更少。
- 适合追求高效率全景分割的开发者,尤其关注计算成本优化者。
全景分割需同时识别可数物体实例与不可数背景区域,对长程上下文建模、多尺度特征表示及高效密集预测提出综合要求。现有卷积与基于Transformer的方法难以同时满足三者:卷积架构难以建模长程依赖,而Transformer方法在高分辨率下计算成本呈二次增长。本文提出MambaPanoptic,一种全Mamba架构的全景分割框架,主要贡献有二:其一,提出MambaFPN,通过Mamba块生成具有全局一致性的多尺度特征表示,计算复杂度为线性;其二,采用类似PanopticFCN的核生成器,统一生成物体与背景核以实现无候选框预测,并引入多阶段QuadMamba特征精炼模块。在Cityscapes与COCO基准测试中,MambaPanoptic在相近模型规模下持续优于PanopticDeepLab与PanopticFCN,且在Cityscapes上性能媲美甚至超越Mask2Former,同时参数量更少。
原文摘要 · Abstract (English)
Panoptic segmentation requires the simultaneous recognition of countable thing instances and amorphous stuff regions, placing joint demands on long-range context modelling, multi-scale feature representation, and efficient dense prediction. Existing convolutional and transformer-based methods struggle to satisfy all three requirements concurrently: convolutional architectures are limited in their capacity to model long-range dependencies, while transformer-based methods incur quadratic computational cost that is prohibitive at high resolutions. In this paper, we propose MambaPanoptic, a fully Mamba-based panoptic segmentation framework that addresses these limitations through two principal contributions. First, we introduce MambaFPN, a top-down feature pyramid that leverages Mamba blocks to generate globally coherent, multi-scale feature representations with linear computational complexity. Second, we adopt a PanopticFCN-style kernel generator that produces unified thing and stuff kernels for proposal-free panoptic prediction, enhanced by a QuadMamba-based feature refinement module applied at multiple network stages. Experiments on the Cityscapes and COCO panoptic segmentation benchmarks demonstrate that MambaPanoptic consistently outperforms PanopticDeepLab and PanopticFCN under comparable model sizes, and matches or surpasses Mask2Former on Cityscapes in PQ and AP while requiring fewer parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。