arXiv:2511.14901cs.CV2025-11被引 1

提出新框架FarSLIP,提升遥感图像细粒度图文对齐能力。

FarSLIP: Discovering Effective CLIP Adaptation for Fine-Grained Remote Sensing Understanding

  • 用局部-局部蒸馏替代全局蒸馏,增强特征区分性。
  • 构建首个多粒度遥感图文数据集MGRS-200k,支持细粒度监督。
  • 不依赖复杂区域对齐,用类别标记实现高效空间感知。

由于CLIP的全局对齐特性限制了其捕捉细粒度细节的能力,现有遥感(RS)专用CLIP变体仍缺乏空间感知。本文指出两大瓶颈:(1) 当前RS图文数据集仅生成整体描述,未充分利用对象级标注;(2) 通用领域中成功的区域-文本对齐方法在遥感数据上应用时常导致性能下降。为此,我们构建首个多粒度遥感图文数据集MGRS-200k,包含丰富的对象级文本监督。进一步研究发现,现有显式区域-文本对齐策略因严重破坏语义一致性而表现不佳。基于此,提出FarSLIP框架:采用局部-局部蒸馏对齐局部与全局视觉线索,提升特征判别力并保留语义连贯性;同时使用简单的CLS token区域-类别对齐,避免复杂像素级对齐。实验表明,FarSLIP在遥感开放词汇语义分割、零样本分类及图文检索等任务上均达新基准。代码与模型已开源。

原文摘要 · Abstract (English)

As CLIP's global alignment limits its ability to capture fine-grained details, recent efforts have focused on enhancing its region-text alignment. However, current remote sensing (RS)-specific CLIP variants still inherit this limited spatial awareness. We identify two key limitations behind this: (1) current RS image-text datasets generate global captions from object-level labels, leaving the original object-level supervision underutilized; (2) despite the success of region-text alignment methods in general domain, their direct application to RS data often leads to performance degradation. To address these, we construct the first multi-granularity RS image-text dataset, MGRS-200k, featuring rich object-level textual supervision for RS region-category alignment. We further investigate existing fine-grained CLIP tuning strategies and find that current explicit region-text alignment methods, whether in a direct or indirect way, underperform due to severe degradation of CLIP's semantic coherence. Building on these, we propose FarSLIP, a Fine-grained Aligned RS Language-Image Pretraining framework. Rather than the commonly used patch-to-CLS self-distillation, FarSLIP employs patch-to-patch distillation to align local and global visual cues, which improves feature discriminability while preserving semantic coherence. Additionally, to effectively utilize region-text supervision, it employs simple CLS token-based region-category alignment rather than explicit patch-level alignment, further enhancing spatial awareness. FarSLIP features improved fine-grained vision-language alignment in RS domain and sets a new state of the art not only on RS open-vocabulary semantic segmentation, but also on image-level tasks such as zero-shot classification and image-text retrieval. Our dataset, code, and models are available at https://github.com/NJU-LHRS/FarSLIP.

遥感理解图文对齐CLIP改进细粒度识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。