用CLIP实现像素级定位，精准分割任意物体。
Repurposing CLIP to Localize at Pixel Level
- 通过追踪CLIP分类过程，提取对象特定注意力区域作为定位线索。
- 在PASCAL和COCO上达到当前最优性能，像素级分割更精准。
- 适合需要零样本分割与高精度定位的研究者使用。
大型视觉语言模型如CLIP在图像级开放集定位方面表现出色，但将其能力拓展至像素级密集预测面临全局特征偏见的挑战。本文提出CLIPix，一种简单有效的框架，将CLIP重新用于像素级定位。通过回溯CLIP的分类过程，CLIPix识别出对象特定的注意力区域，并将其重用为像素级定位信号。为缓解全局偏见引入的噪声，我们设计了抗噪修正策略，优化这些信号以提升分割精度。此外，提出定位嵌入策略，融合定位信息与丰富细节，实现高分辨率精确分割。该方法保持CLIP的泛化能力，解锁其对任意物体的分割潜力。在PASCAL和COCO数据集上的大量实验表明，CLIPix达到当前最佳性能，验证其有效性。
原文摘要 · Abstract (English)
Large-scale Vision-Language Models like CLIP have demonstrated impressive open-set localization capabilities at the image level. However, adapting this capability to pixel-level dense prediction poses challenges due to global feature biases. In this paper, we introduce CLIPix, a simple yet effective framework that repurposes CLIP to perform pixel-level localization. By tracing back CLIP's classification process, CLIPix identifies object-specific attentive regions and repurposes them as pixel-level localization cues. To address noise introduced by global biases, we propose a Noise-Resistant Correction strategy, refining these cues for more precise segmentation. Additionally, we introduce a Localization Embedding strategy to integrate both localization and enriched detail information, enabling accurate, high-resolution segmentation. Our approach preserves CLIP's generalization strength and unlocks its potential for segmenting arbitrary objects. Extensive experiments on the PASCAL and COCO datasets demonstrate that CLIPix achieves state-of-the-art performance, underscoring its effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。