提升CLIP的空间感知能力,让模型更懂图像中物体的位置关系。
Refining CLIP's Spatial Awareness: A Visual-Centric Perspective
- 用空间相关性蒸馏保留CLIP原有空间结构
- 引入轻量级重构器提取高质量密集特征
- 在开放词汇密集预测任务上达到顶尖性能
对比语言-图像预训练(CLIP)在全局语义对齐上表现优异,但在空间信息敏感度上不足,导致其在需要精确空间理解的任务中表现不佳。尽管区域-语言对齐(RLA)方法能提升密集多模态任务性能,但经RLA微调的CLIP ViT会显著损失空间感知能力。为此,我们提出空间相关性蒸馏(SCD)框架,以保持CLIP固有的空间结构并缓解这一退化。为进一步增强空间相关性,我们设计了一个轻量级重构器,直接从原始CLIP中提取精细的相关性特征输入SCD。二者协同构建出稳健的蒸馏机制,使CLIP ViT可同时融合视觉-语言与视觉中心改进,在多个开放词汇密集预测基准上实现最优表现。
原文摘要 · Abstract (English)
Contrastive Language-Image Pre-training (CLIP) excels in global alignment with language but exhibits limited sensitivity to spatial information, leading to strong performance in zero-shot classification tasks but underperformance in tasks requiring precise spatial understanding. Recent approaches have introduced Region-Language Alignment (RLA) to enhance CLIP's performance in dense multimodal tasks by aligning regional visual representations with corresponding text inputs. However, we find that CLIP ViTs fine-tuned with RLA suffer from notable loss in spatial awareness, which is crucial for dense prediction tasks. To address this, we propose the Spatial Correlation Distillation (SCD) framework, which preserves CLIP's inherent spatial structure and mitigates the above degradation. To further enhance spatial correlations, we introduce a lightweight Refiner that extracts refined correlations directly from CLIP before feeding them into SCD, based on an intriguing finding that CLIP naturally captures high-quality dense features. Together, these components form a robust distillation framework that enables CLIP ViTs to integrate both visual-language and visual-centric improvements, achieving state-of-the-art results across various open-vocabulary dense prediction benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。