arXiv:2412.04533cs.CV2024-12CVPR被引 26

改进开放词汇分割中的掩码分类,提升准确率。

Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation

  • 用语义激活图替代原始掩码,增强上下文信息
  • 引入掩码一致性损失,提升对不同掩码的鲁棒性
  • 可即插即用,适配主流开放词汇分割方法

当前开放词汇分割方法通过掩码生成器预测分割掩码,并利用预训练视觉语言模型(如CLIP)通过掩码池化进行分类。尽管效果良好,但一个反直觉现象是:高精度掩码反而难以通过池化其区域内的CLIP图像嵌入获得准确分类结果。本文揭示了掩码池化的性能瓶颈,提出Mask-Adapter——一种简单有效的解决方案。相比直接使用提议掩码,该方法从提议掩码中提取语义激活图,提供更丰富的上下文信息,并确保掩码与CLIP之间的对齐。此外,提出掩码一致性损失,促使相似交并比(IoU)的掩码获得相近的CLIP嵌入,增强模型对不同预测掩码的鲁棒性。Mask-Adapter可无缝集成至基于掩码池化的开放词汇分割方法中,以即插即用方式实现更准确的分类。在多个零样本基准测试中,该方法显著提升了现有主流方法的性能。值得注意的是,其亦成功扩展至SAM,在多个开放词汇分割数据集上取得优异结果。

原文摘要 · Abstract (English)

Recent open-vocabulary segmentation methods adopt mask generators to predict segmentation masks and leverage pre-trained vision-language models, e.g., CLIP, to classify these masks via mask pooling. Although these approaches show promising results, it is counterintuitive that accurate masks often fail to yield accurate classification results through pooling CLIP image embeddings within the mask regions. In this paper, we reveal the performance limitations of mask pooling and introduce Mask-Adapter, a simple yet effective method to address these challenges in open-vocabulary segmentation. Compared to directly using proposal masks, our proposed Mask-Adapter extracts semantic activation maps from proposal masks, providing richer contextual information and ensuring alignment between masks and CLIP. Additionally, we propose a mask consistency loss that encourages proposal masks with similar IoUs to obtain similar CLIP embeddings to enhance models' robustness to varying predicted masks. Mask-Adapter integrates seamlessly into open-vocabulary segmentation methods based on mask pooling in a plug-and-play manner, delivering more accurate classification results. Extensive experiments across several zero-shot benchmarks demonstrate significant performance gains for the proposed Mask-Adapter on several well-established methods. Notably, Mask-Adapter also extends effectively to SAM and achieves impressive results on several open-vocabulary segmentation datasets. Code and models are available at https://github.com/hustvl/MaskAdapter.

开放词汇分割掩码池化CLIP即插即用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。