用真实标注掩码提升视觉语言对齐，让模型分割更准
High-Quality Mask Tuning Matters for Open-Vocabulary Segmentation
- 用真实掩码替代生成掩码进行微调，增强语义对齐
- 在多个数据集上提升1.7到3.1的mIoU，最高增益达3.1
- 适合做开放词汇图像分割的研究者和应用开发者
开放词汇图像分割依赖掩码生成器与视觉语言模型（如CLIP）的协同。以往方法侧重生成掩码并训练时对齐掩码特征与文本嵌入。本文发现，使用低质量生成掩码会削弱区域表示中视觉与语言的对齐。为此提出新微调框架MaskCLIP++，采用真实掩码而非生成掩码来增强CLIP的掩码分类能力。由于带掩码标注的数据集多样性有限，引入一致性对齐原则缓解细粒度分类偏差。低成本微调后，MaskCLIP++显著提升多领域数据集上的掩码分类性能：在A-847、PC-459、A-150、PC-59和PAS-20数据集上分别取得+1.7、+2.3、+2.1、+3.1和+0.3 mIoU的改进。代码已公开于https://github.com/HVision-NKU/MaskCLIPpp。
原文摘要 · Abstract (English)
Open-vocabulary image segmentation has been advanced through the synergy between mask generators and vision-language models like Contrastive Language-Image Pre-training (CLIP). Previous approaches focus on generating masks while aligning mask features with text embeddings during training. In this paper, we observe that relying on generated low-quality masks can weaken the alignment of vision and language in regional representations. This motivates us to present a new fine-tuning framework, named MaskCLIP++, which uses ground-truth masks instead of generated masks to enhance the mask classification capability of CLIP. Due to the limited diversity of image segmentation datasets with mask annotations, we propose incorporating a consistency alignment principle during fine-tuning, which alleviates categorical bias toward the fine-tuning dataset. After low-cost fine-tuning, MaskCLIP++ significantly improves the mask classification performance on multi-domain datasets. Combining with the mask generator in previous state-of-the-art mask-based open vocabulary segmentation methods, we achieve performance improvements of +1.7, +2.3, +2.1, +3.1, and +0.3 mIoU on the A-847, PC-459, A-150, PC-59, and PAS-20 datasets, respectively. Code is avaliable at https://github.com/HVision-NKU/MaskCLIPpp .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。