为遥感图像设计双模态提示学习,提升视觉语言模型泛化能力
bi-modal textual prompt learning for vision-language models in remote sensing
- 用图像描述生成文本摘要,与视觉特征融合构建双模提示
- 在4个遥感数据集上平均提升2%,显著优于现有方法
- 适合遥感领域少样本任务,轻量级设计易部署
提示学习(PL)已成为在有限监督下适配视觉语言模型(如CLIP)的高效策略。尽管在自然图像数据集上表现优异,其在遥感(RS)图像中的迁移能力仍待探索。遥感数据具有多标签场景、类内差异大、分辨率多样等挑战,导致现有提示方法难以识别主导语义线索,且泛化能力弱。为此,我们提出针对遥感任务的轻量级双模态提示学习框架BiMoRS。该方法利用冻结的图像描述模型(如BLIP-2)生成遥感图像的文本语义摘要,经BERT分词后与CLIP提取的高层视觉特征融合,通过轻量级交叉注意力模块将可学习查询提示条件化于融合表示,从而生成上下文感知提示,无需修改CLIP主干。我们在四个遥感数据集上评估了三种域泛化任务,结果表明性能持续提升,平均优于强基线2%。代码已开源:https://github.com/ipankhi/BiMoRS。
原文摘要 · Abstract (English)
Prompt learning (PL) has emerged as an effective strategy to adapt vision-language models (VLMs), such as CLIP, for downstream tasks under limited supervision. While PL has demonstrated strong generalization on natural image datasets, its transferability to remote sensing (RS) imagery remains underexplored. RS data present unique challenges, including multi-label scenes, high intra-class variability, and diverse spatial resolutions, that hinder the direct applicability of existing PL methods. In particular, current prompt-based approaches often struggle to identify dominant semantic cues and fail to generalize to novel classes in RS scenarios. To address these challenges, we propose BiMoRS, a lightweight bi-modal prompt learning framework tailored for RS tasks. BiMoRS employs a frozen image captioning model (e.g., BLIP-2) to extract textual semantic summaries from RS images. These captions are tokenized using a BERT tokenizer and fused with high-level visual features from the CLIP encoder. A lightweight cross-attention module then conditions a learnable query prompt on the fused textual-visual representation, yielding contextualized prompts without altering the CLIP backbone. We evaluate BiMoRS on four RS datasets across three domain generalization (DG) tasks and observe consistent performance gains, outperforming strong baselines by up to 2% on average. Codes are available at https://github.com/ipankhi/BiMoRS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。