构建遥感图文双粒度数据集与模型，提升图文对齐精度
DGTRSD & DGTRS-CLIP: A Dual-Granularity Remote Sensing Image-Text Dataset and Vision Language Foundation Model for Alignment
- 设计双粒度图文数据集，每图配短描述与长文本
- 提出课程学习框架，融合长短文本实现语义对齐
- 在4项零样本任务中均超越现有方法，适合遥感多模态研究
基于CLIP架构的遥感视觉语言基础模型通常依赖短文本描述，导致语义表征不完整。尽管长文本包含更丰富信息，但现有模型受限于文本编码能力，难以有效处理，且缺乏同时关联短文本和长文本的资源。为此，我们提出DGTRSD双粒度遥感图像-文本数据集，每个图像均配有短文本描述和长文本说明，为双粒度语义建模提供基础。基于此，我们进一步提出DGTRS-CLIP，一种结合短文本与长文本监督的双粒度课程学习框架，实现双粒度语义对齐。在四个典型零样本任务（长文本跨模态检索、短文本跨模态检索、图像分类、语义定位）上的大量实验表明，DGTRS-CLIP在所有任务中均持续优于现有方法。代码已开源，地址：https://github.com/MitsuiChen14/DGTRS。
原文摘要 · Abstract (English)
Vision Language Foundation Models based on CLIP architecture for remote sensing primarily rely on short text captions, which often result in incomplete semantic representations. Although longer captions convey richer information, existing models struggle to process them effectively because of limited text-encoding capacity, and there remains a shortage of resources that align remote sensing images with both short text and long text captions. To address this gap, we introduce DGTRSD, a dual-granularity remote sensing image-text dataset, where each image is paired with both a short text caption and a long text description, providing a solid foundation for dual-granularity semantic modeling. Based on this, we further propose DGTRS-CLIP, a dual-granularity curriculum learning framework that combines short text and long text supervision to achieve dual-granularity semantic alignment. Extensive experiments on four typical zero-shot tasks: long text cross-modal retrieval, short text cross-modal retrieval, image classification, and semantic localization demonstrate that DGTRS-CLIP consistently outperforms existing methods across all tasks. The code has been open-sourced and is available at https://github.com/MitsuiChen14/DGTRS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。