用高分辨率图像+SAM提升CLIP分割性能,无需训练。
Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation
- 先拼接子图特征,再用SAM建模全局关联
- 在8个基准上mIoU从44.4提升至48.6
- 无需训练,适合快速部署的开放词汇分割
尽管对比语言-图像预训练(CLIP)推动了开放词汇预测的发展,其在语义分割上的表现仍不理想,主要源于空间不变的语义特征和受限的分辨率。以往方法通过修改CLIP图像编码器的自注意力机制缓解空间不变性问题,但分辨率限制仍未解决。不同于以往先分割后拼接的滑动窗口方法,本文提出先拼接后分割的新范式,利用分割任意模型(SAM)从高分辨率图像中提取细粒度语义关联。具体地,我们提出Trident框架,该框架不需训练,先融合CLIP与DINO提取的子图特征,再通过SAM编码器构建相关性矩阵实现全局聚合,从而扩大感受野,实现高效分割。此外,我们设计一种优化策略,将CLIP粗分割结果转为SAM提示,进一步提升性能。Trident在8个基准上相比当前最优方法显著提升mIoU,从44.4增至48.6。代码已开源。
原文摘要 · Abstract (English)
While Contrastive Language-Image Pre-training (CLIP) has advanced open-vocabulary predictions, its performance on semantic segmentation remains suboptimal. This shortfall primarily stems from its spatial-invariant semantic features and constrained resolution. While previous adaptations addressed spatial invariance semantic by modifying the self-attention in CLIP's image encoder, the issue of limited resolution remains unexplored. Different from previous segment-then-splice methods that segment sub-images via a sliding window and splice the results, we introduce a splice-then-segment paradigm that incorporates Segment-Anything Model (SAM) to tackle the resolution issue since SAM excels at extracting fine-grained semantic correlations from high-resolution images. Specifically, we introduce Trident, a training-free framework that first splices features extracted by CLIP and DINO from sub-images, then leverages SAM's encoder to create a correlation matrix for global aggregation, enabling a broadened receptive field for effective segmentation. Besides, we propose a refinement strategy for CLIP's coarse segmentation outputs by transforming them into prompts for SAM, further enhancing the segmentation performance. Trident achieves a significant improvement in the mIoU across eight benchmarks compared with the current SOTA, increasing from 44.4 to 48.6.Code is available at https://github.com/YuHengsss/Trident.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。