解决图像文本匹配中的模糊与高阶关系问题,提升跨模态理解精度。
Ambiguity-Aware and High-Order Relation Learning for Multi-Grained Image-Text Matching
- 通过动态聚类原型对比学习构建统一表征空间,缓解软正样本干扰。
- 在多个数据集上超越现有方法,Flickr30K的R@1达84.3%。
- 适合需要细粒度图文对齐的研究者,尤其关注语义模糊与关系建模场景。
图像-文本匹配对于弥合计算机视觉与自然语言处理之间的语义鸿沟至关重要。然而,现有方法在处理相似实例间的高阶关联和语义模糊性方面仍面临挑战。这些模糊性源于软正样本(语义相似但标注错误)与软负样本(局部匹配但全局不一致)之间的细微差异,导致匹配不确定性。此外,当前方法未能充分利用训练批次内语义相似实例间的邻域关系,限制了模型对高阶共享知识的学习能力。本文提出模糊感知与高阶关系学习框架(AAHR),通过动态聚类原型对比学习构建统一表征空间,有效缓解软正样本问题。框架引入全局与局部特征提取机制及自适应聚合网络,显著增强全粒度语义理解能力。同时,利用跨模态与同模态相关矩阵深入分析样本间邻域关系,并结合图神经网络强化实例间语义交互。此外,采用动量对比学习扩展负样本集合。上述策略协同提升模型区分能力。实验表明,AAHR在Flickr30K、MSCOCO和ECCV Caption数据集上均优于现有最优方法,显著提高图像-文本匹配的准确率与效率。代码与模型检查点见https://github.com/Image-Text-Matching/AAHR。
原文摘要 · Abstract (English)
Image-text matching is crucial for bridging the semantic gap between computer vision and natural language processing. However, existing methods still face challenges in handling high-order associations and semantic ambiguities among similar instances. These ambiguities arise from subtle differences between soft positive samples (semantically similar but incorrectly labeled) and soft negative samples (locally matched but globally inconsistent), creating matching uncertainties. Furthermore, current methods fail to fully utilize the neighborhood relationships among semantically similar instances within training batches, limiting the model's ability to learn high-order shared knowledge. This paper proposes the Ambiguity-Aware and High-order Relation learning framework (AAHR) to address these issues. AAHR constructs a unified representation space through dynamic clustering prototype contrastive learning, effectively mitigating the soft positive sample problem. The framework introduces global and local feature extraction mechanisms and an adaptive aggregation network, significantly enhancing full-grained semantic understanding capabilities. Additionally, AAHR employs intra-modal and inter-modal correlation matrices to investigate neighborhood relationships among sample instances thoroughly. It incorporates GNN to enhance semantic interactions between instances. Furthermore, AAHR integrates momentum contrastive learning to expand the negative sample set. These combined strategies significantly improve the model's ability to discriminate between features. Experimental results demonstrate that AAHR outperforms existing state-of-the-art methods on Flickr30K, MSCOCO, and ECCV Caption datasets, considerably improving the accuracy and efficiency of image-text matching. The code and model checkpoints for this research are available at https://github.com/Image-Text-Matching/AAHR .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。