arXiv:2601.18190cs.CV2026-01

用关键词引导局部图像匹配,提升遥感图文检索精度

Multi-Perspective Subimage CLIP with Keyword Guidance for Remote Sensing Image-Text Retrieval

  • 用大模型提取关键词,指导分割生成细粒度视角
  • 仅微调少量参数,实现比全量微调更高的检索准确率
  • 适合需要高效适配遥感数据的科研与工程人员

视觉-语言预训练模型如CLIP显著推动了遥感图文检索(RSITR)发展。然而,现有方法多依赖粗粒度全局对齐,常忽略高空影像中密集、多尺度的语义信息。此外,通过全量微调适配这些重型模型会带来高昂计算成本并可能导致灾难性遗忘。为此,我们提出MPS-CLIP,一种参数高效的框架,旨在将检索范式从全局匹配转向关键词引导的细粒度对齐。具体地,利用大语言模型(LLM)提取核心语义关键词,引导分割任意模型(SamGeo)生成语义相关子视角。为高效适配冻结主干网络,引入门控全局注意力(G^2A)适配器,以极小开销捕捉全局上下文与长程依赖。同时,多视角表示(MPR)模块将这些局部线索聚合为鲁棒的多视角嵌入。框架通过结合多视角对比损失与加权三元组损失的混合目标进行优化,动态选择响应最强的视角以抑制噪声并强化精确语义匹配。在RSICD和RSITMD基准上的大量实验表明,MPS-CLIP分别达到35.18%和48.40%的均值召回率(mR),显著优于全量微调基线及近期先进方法。代码已开源:https://github.com/Lcrucial1f/MPS-CLIP。

原文摘要 · Abstract (English)

Vision-Language Pre-training (VLP) models like CLIP have significantly advanced Remote Sensing Image-Text Retrieval (RSITR). However, existing methods predominantly rely on coarse-grained global alignment, which often overlooks the dense, multi-scale semantics inherent in overhead imagery. Moreover, adapting these heavy models via full fine-tuning incurs prohibitive computational costs and risks catastrophic forgetting. To address these challenges, we propose MPS-CLIP, a parameter-efficient framework designed to shift the retrieval paradigm from global matching to keyword-guided fine-grained alignment. Specifically, we leverage a Large Language Model (LLM) to extract core semantic keywords, guiding the Segment Anything Model (SamGeo) to generate semantically relevant sub-perspectives. To efficiently adapt the frozen backbone, we introduce a Gated Global Attention (G^2A) adapter, which captures global context and long-range dependencies with minimal overhead. Furthermore, a Multi-Perspective Representation (MPR) module aggregates these local cues into robust multi-perspective embeddings. The framework is optimized via a hybrid objective combining multi-perspective contrastive and weighted triplet losses, which dynamically selects maximum-response perspectives to suppress noise and enforce precise semantic matching. Extensive experiments on the RSICD and RSITMD benchmarks demonstrate that MPS-CLIP achieves state-of-the-art performance with 35.18% and 48.40% mean Recall (mR), respectively, significantly outperforming full fine-tuning baselines and recent competitive methods. Code is available at https://github.com/Lcrucial1f/MPS-CLIP.

遥感图像图文检索多视角参数高效

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。