arXiv:2504.03193cs.CV2025-04CVPR被引 36

用Mamba融合视觉与文本模型,提升跨域语义分割精度。

Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation

  • 基于Mamba构建双模型融合框架,兼顾序列与空间动态
  • 在合成到真实场景达68.20 mIoU,真实到真实达71.87 mIoU
  • 适合关注多模态融合与高效长序列建模的研究者

视觉基础模型(VFMs)和视觉语言模型(VLMs)因其强大的泛化能力,在跨域语义分割(DGSS)中备受关注。然而现有方法通常仅依赖单一模型,忽视了它们的互补优势:VFMs(如DINOv2)擅长捕捉细粒度特征,而VLMs(如CLIP)虽具备强文本对齐能力,却难以处理粗粒度信息。将两者通过注意力机制融合面临挑战,因增加的图像块令牌使长序列建模复杂化。为此,我们提出MFuser,一种基于Mamba的融合框架,能高效结合两者优势并保持线性序列扩展性。MFuser包含两个核心组件:MVFuser作为联合适配器,同时捕捉序列与空间动态;MTEnhancer为混合注意力-Mamba模块,通过引入图像先验优化文本嵌入。该方法实现精确的局部特征表达与强文本对齐,且计算开销低。大量实验表明,MFuser显著优于当前最优方法,在合成到真实场景达到68.20 mIoU,真实到真实场景达71.87 mIoU。代码已开源于https://github.com/devinxzhang/MFuser。

原文摘要 · Abstract (English)

Vision Foundation Models (VFMs) and Vision-Language Models (VLMs) have gained traction in Domain Generalized Semantic Segmentation (DGSS) due to their strong generalization capabilities. However, existing DGSS methods often rely exclusively on either VFMs or VLMs, overlooking their complementary strengths. VFMs (e.g., DINOv2) excel at capturing fine-grained features, while VLMs (e.g., CLIP) provide robust text alignment but struggle with coarse granularity. Despite their complementary strengths, effectively integrating VFMs and VLMs with attention mechanisms is challenging, as the increased patch tokens complicate long-sequence modeling. To address this, we propose MFuser, a novel Mamba-based fusion framework that efficiently combines the strengths of VFMs and VLMs while maintaining linear scalability in sequence length. MFuser consists of two key components: MVFuser, which acts as a co-adapter to jointly fine-tune the two models by capturing both sequential and spatial dynamics; and MTEnhancer, a hybrid attention-Mamba module that refines text embeddings by incorporating image priors. Our approach achieves precise feature locality and strong text alignment without incurring significant computational overhead. Extensive experiments demonstrate that MFuser significantly outperforms state-of-the-art DGSS methods, achieving 68.20 mIoU on synthetic-to-real and 71.87 mIoU on real-to-real benchmarks. The code is available at https://github.com/devinxzhang/MFuser.

跨域分割多模态融合Mamba

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。