解决遥感图文检索中的错配数据问题,提升模型鲁棒性。
Robust Remote Sensing Image-Text Retrieval with Noisy Correspondence
- 设计自适应学习策略,按难易程度逐步训练带噪声的数据对。
- 在高噪声率下,性能超越现有方法,最高提升12.3%。
- 适合处理真实场景中难以获取精准标注的遥感图文数据。
遥感图像-文本检索(RSITR)作为连接遥感视觉与语言理解的关键任务,近年来受到广泛关注。然而,几乎所有现有方法都隐含假设图像与文本配对完全准确。实际上,获取大规模精准对齐数据代价高昂甚至不可行。此外,我们发现遥感数据集(如RSITMD)中存在大量不准确或错配的图文描述。基于此,我们揭示了RSITR中一个长期被忽视的问题——噪声对应(Noisy Correspondence, NC)。为此,提出一种新的鲁棒遥感图像-文本检索框架(RRSITR),采用自适应学习策略模拟人类认知过程,从易到难逐步学习多模态噪声数据。具体地,根据每对样本的损失大小将其分为三类:干净对、模糊对和噪声对;通过损失值动态赋权,评估每对样本的可靠性;并设计新型多模态自适应学习函数,动态调节训练顺序与权重,实现渐进式学习。针对噪声对,引入一种可动态调整软边距的鲁棒三元组损失,增强对噪声的抵抗能力。在三个主流基准数据集上的大量实验表明,所提方法在高噪声率下显著优于当前最优方法,尤其在噪声率高达40%时仍保持优异性能。代码已开源:https://github.com/MSFLabX/RRSITR
原文摘要 · Abstract (English)
As a pivotal task that bridges remote visual and linguistic understanding, Remote Sensing Image-Text Retrieval (RSITR) has attracted considerable research interest in recent years. However, almost all RSITR methods implicitly assume that image-text pairs are matched perfectly. In practice, acquiring a large set of well-aligned data pairs is often prohibitively expensive or even infeasible. In addition, we also notice that the remote sensing datasets (e.g., RSITMD) truly contain some inaccurate or mismatched image text descriptions. Based on the above observations, we reveal an important but untouched problem in RSITR, i.e., Noisy Correspondence (NC). To overcome these challenges, we propose a novel Robust Remote Sensing Image-Text Retrieval (RRSITR) paradigm that designs a self-paced learning strategy to mimic human cognitive learning patterns, thereby learning from easy to hard from multi-modal data with NC. Specifically, we first divide all training sample pairs into three categories based on the loss magnitude of each pair, i.e., clean sample pairs, ambiguous sample pairs, and noisy sample pairs. Then, we respectively estimate the reliability of each training pair by assigning a weight to each pair based on the values of the loss. Further, we respectively design a new multi-modal self-paced function to dynamically regulate the training sequence and weights of the samples, thus establishing a progressive learning process. Finally, for noisy sample pairs, we present a robust triplet loss to dynamically adjust the soft margin based on semantic similarity, thereby enhancing the robustness against noise. Extensive experiments on three popular benchmark datasets demonstrate that the proposed RRSITR significantly outperforms the state-of-the-art methods, especially in high noise rates. The code is available at: https://github.com/MSFLabX/RRSITR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。