用文字查询遥感影像时间序列,实现跨模态精准检索。
Self-Supervised Cross-Modal Text-Image Time Series Retrieval in Remote Sensing
- 自监督学习构建图文时间序列联合嵌入空间
- 双融合策略有效捕捉影像时间变化特征
- 适合遥感场景下无图像查询的跨模态检索
遥感图像时间序列检索(ITSR)研究日益受到关注。给定用户定义的图像时间序列(即查询序列),现有方法从大规模档案中检索内容相似的序列。然而,当前方法多为单模态检索,假设用户始终可提供目标模态的查询序列,这在实际操作中难以满足。为此,首次在遥感领域提出跨模态文本-图像时间序列检索(text-ITSR)任务。本文提出一种自监督跨模态text-ITSR方法,支持以文本句或图像序列作为查询进行双向检索。聚焦于成对影像(双时相影像),方法包含两个核心组件:1)模态专用编码器,提取双时相影像与文本句子的判别性语义特征;2)模态专用投影头,将文本与图像表示映射至共享嵌入空间。为建模双时相影像的时间信息,采用两种融合策略:全局特征融合(GFF)通过简单有效的算子整合全局特征;基于Transformer的特征融合(TFF)实现细粒度的时间集成。在两个基准遥感数据集上的大量实验表明,该方法能准确检索与查询文本句(或双时相影像)语义相关的双时相影像(或文本句)。代码已公开于 https://git.tu-berlin.de/rsim/cross-modal-text-tsir。
原文摘要 · Abstract (English)
The development of image time series retrieval (ITSR) methods is a growing research interest in remote sensing (RS). Given a user-defined image time series (i.e., the query time series), ITSR methods search and retrieve from large archives the image time series that have similar content to the query time series. Existing ITSR methods in RS are designed for unimodal retrieval problems, relying on an assumption that users always have access to a query image time series in the considered image modality. In operational scenarios, this assumption may not hold. To overcome this issue, as a first time in RS we introduce the task of cross-modal text-image time series retrieval (text-ITSR). In detail, we present a self-supervised cross-modal text-ITSR method that enables the retrieval of image time series using text sentences as queries, and vice versa. We focus our attention on text-ITSR in pairs of images (i.e., bitemporal images). Our text-ITSR method consists of two key components: 1) modality-specific encoders to model the semantic content of bitemporal images and text sentences with discriminative features; and 2) modality-specific projection heads to align textual and image representations in a shared embedding space. To effectively model the temporal information in the bitemporal images, we exploit two fusion strategies: i) global feature fusion (GFF) strategy that combines global image features through simple yet effective operators; and ii) transformer-based feature fusion (TFF) strategy that leverages transformers for fine-grained temporal integration. Extensive experiments conducted on two benchmark RS archives demonstrate the effectiveness of our method in accurately retrieving semantically relevant bitemporal images (or text sentences) to a query text sentence (or bitemporal image). The code of this work is publicly available at https://git.tu-berlin.de/rsim/cross-modal-text-tsir .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。