解决跨模态检索中查询分布变化导致的性能下降问题
Test-time Adaptation for Cross-modal Retrieval with Query Shift
- 提出测试时自适应模块，动态优化查询预测结果
- 在多个数据集上实现显著性能提升，尤其在分布外查询场景
- 适合实际应用中查询多样性高的跨模态检索系统
现有跨模态检索方法大多依赖于查询分布与源域一致的假设，但在真实场景中，查询复杂多变，常出现查询分布偏移（query shift）问题。本文观察到，查询分布偏移会降低查询模态内部的一致性（即模态内散度），并加剧查询与图像库之间模态间的差距。基于此，提出一种名为测试时跨模态检索自适应（TCR）的新方法：通过引入一个查询预测修正模块和联合优化目标，抑制查询偏移对共享表示空间的干扰，实现模型在在线查询流中的自适应能力。大量实验验证了TCR在应对查询分布偏移方面的有效性。代码将在论文被接受后发布。
原文摘要 · Abstract (English)
The success of most existing cross-modal retrieval methods heavily relies on the assumption that the given queries follow the same distribution of the source domain. However, such an assumption is easily violated in real-world scenarios due to the complexity and diversity of queries, thus leading to the query shift problem. Specifically, query shift refers to the online query stream originating from the domain that follows a different distribution with the source one. In this paper, we observe that query shift would not only diminish the uniformity (namely, within-modality scatter) of the query modality but also amplify the gap between query and gallery modalities. Based on the observations, we propose a novel method dubbed Test-time adaptation for Cross-modal Retrieval (TCR). In brief, TCR employs a novel module to refine the query predictions (namely, retrieval results of the query) and a joint objective to prevent query shift from disturbing the common space, thus achieving online adaptation for the cross-modal retrieval models with query shift. Expensive experiments demonstrate the effectiveness of the proposed TCR against query shift. The code will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。