用最大匹配优化跨模态检索，防止表示坍塌
Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval
- 采用多向量集合表示，增强跨模态关系捕捉能力
- 在MS-COCO和Flickr30k上达到当前最佳性能
- 适合追求鲁棒性与语义多样性的检索任务
跨模态图像-文本检索因不同模态间存在多样的潜在关联而具有挑战性。传统方法使用单一向量表示每个样本的语义，难以捕捉模态间的细微和多样化关系。基于集合的方法通过为每个样本分配多个嵌入向量，提供了更丰富的表示方式。然而，这类方法仍面临监督稀疏和集合坍塌问题，限制了其效果。本文提出最大配对赋值相似性（Maximal Pair Assignment Similarity），以优化嵌入集合间的一一对应匹配，保持集合内部的语义多样性。同时引入两种损失函数：全局判别损失提升嵌入间的区分度，集合内发散损失防止集合内部坍塌。该方法在不依赖外部数据的情况下，在MS-COCO和Flickr30k数据集上达到当前最优性能。
原文摘要 · Abstract (English)
Cross-modal image-text retrieval is challenging because of the diverse possible associations between content from different modalities. Traditional methods learn a single-vector embedding to represent semantics of each sample, but struggle to capture nuanced and diverse relationships that can exist across modalities. Set-based approaches, which represent each sample with multiple embeddings, offer a promising alternative, as they can capture richer and more diverse relationships. In this paper, we show that, despite their promise, these set-based representations continue to face issues including sparse supervision and set collapse, which limits their effectiveness. To address these challenges, we propose Maximal Pair Assignment Similarity to optimize one-to-one matching between embedding sets which preserve semantic diversity within the set. We also introduce two loss functions to further enhance the representations: Global Discriminative Loss to enhance distinction among embeddings, and Intra-Set Divergence Loss to prevent collapse within each set. Our method achieves state-of-the-art performance on MS-COCO and Flickr30k without relying on external data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。