提出双库Sinkhorn归一化,有效缓解跨模态检索中的枢纽问题。
Hubness Reduction with Dual Bank Sinkhorn Normalization for Cross-Modal Retrieval
- 通过平衡查询与目标概率,改进传统归一化方法
- 在图像-文本、视频-文本、音频-文本任务中提升检索精度
- 适用于需高精度跨模态匹配的场景,如多模态搜索
过去十年间,跨模态检索取得了显著进展,相似性度量日益精准。然而,枢纽问题(即少数目标频繁成为大量查询的最近邻)仍严重制约相似性测量精度。尽管已有多种缓解方法,其内在机制仍不清晰。本文分析了广泛使用的Inverted Softmax方法,揭示其在检索中平衡目标概率的有效性。基于此,提出概率平衡框架,并进一步引入Sinkhorn归一化(SN)以同时平衡查询与目标概率。针对真实查询分布未知的情况,指出仅用查询库估计目标枢纽性会导致显著分布偏差,进而影响性能。为此,提出双库Sinkhorn归一化(DBSN),引入对应的目标库以缩小分布差距。在图像-文本、视频-文本、音频-文本等跨模态检索任务上的全面评估显示,DBSN持续提升性能。代码已开源:https://github.com/ppanzx/DBSN。
原文摘要 · Abstract (English)
The past decade has witnessed rapid advancements in cross-modal retrieval, with significant progress made in accurately measuring the similarity between cross-modal pairs. However, the persistent hubness problem, a phenomenon where a small number of targets frequently appear as nearest neighbors to numerous queries, continues to hinder the precision of similarity measurements. Despite several proposed methods to reduce hubness, their underlying mechanisms remain poorly understood. To bridge this gap, we analyze the widely-adopted Inverted Softmax approach and demonstrate its effectiveness in balancing target probabilities during retrieval. Building on these insights, we propose a probability-balancing framework for more effective hubness reduction. We contend that balancing target probabilities alone is inadequate and, therefore, extend the framework to balance both query and target probabilities by introducing Sinkhorn Normalization (SN). Notably, we extend SN to scenarios where the true query distribution is unknown, showing that current methods, which rely solely on a query bank to estimate target hubness, produce suboptimal results due to a significant distributional gap between the query bank and targets. To mitigate this issue, we introduce Dual Bank Sinkhorn Normalization (DBSN), incorporating a corresponding target bank alongside the query bank to narrow this distributional gap. Our comprehensive evaluation across various cross-modal retrieval tasks, including image-text retrieval, video-text retrieval, and audio-text retrieval, demonstrates consistent performance improvements, validating the effectiveness of both SN and DBSN. All codes are publicly available at https://github.com/ppanzx/DBSN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。