arXiv:2502.19962cs.CVcs.IR2025-02CVPR被引 7

通过关系一致性提升跨模态匹配准确率,有效过滤错误配对。

ReCon: Enhancing True Correspondence Discrimination through Relation Consistency for Robust Noisy Correspondence Learning

  • 引入跨模态与模内关系一致性双约束机制
  • 在三个数据集上显著优于现有最优方法
  • 适合需要高精度图像文本匹配的场景

从包含错误配对的多模态数据集中能否准确识别真实对应关系?现有方法主要关注跨模态表示间的相似性匹配,可能忽略模态内部的关系一致性,而这对区分真实与虚假对应关系至关重要。这种忽略常导致将负样本误判为正样本,引发性能下降。为此,我们提出通用的关系一致性学习框架 ReCon,以更准确地鉴别多模态数据中的真实对应关系,有效缓解错配带来的负面影响。ReCon 利用新颖的关系一致性学习,实现跨模态关系一致性和模内关系一致性的双重对齐。得益于这种双重关系约束,ReCon 显著提升了真实对应关系的判别能力,从而可靠地过滤错误配对,降低错误监督风险。在 Flickr30K、MS-COCO 和 Conceptual Captions 三个常用基准数据集上的大量实验表明,ReCon 在效果和优越性上均优于其他 SOTA 方法。代码已开源:https://github.com/qxzha/ReCon。

原文摘要 · Abstract (English)

Can we accurately identify the true correspondences from multimodal datasets containing mismatched data pairs? Existing methods primarily emphasize the similarity matching between the representations of objects across modalities, potentially neglecting the crucial relation consistency within modalities that are particularly important for distinguishing the true and false correspondences. Such an omission often runs the risk of misidentifying negatives as positives, thus leading to unanticipated performance degradation. To address this problem, we propose a general Relation Consistency learning framework, namely ReCon, to accurately discriminate the true correspondences among the multimodal data and thus effectively mitigate the adverse impact caused by mismatches. Specifically, ReCon leverages a novel relation consistency learning to ensure the dual-alignment, respectively of, the cross-modal relation consistency between different modalities and the intra-modal relation consistency within modalities. Thanks to such dual constrains on relations, ReCon significantly enhances its effectiveness for true correspondence discrimination and therefore reliably filters out the mismatched pairs to mitigate the risks of wrong supervisions. Extensive experiments on three widely-used benchmark datasets, including Flickr30K, MS-COCO, and Conceptual Captions, are conducted to demonstrate the effectiveness and superiority of ReCon compared with other SOTAs. The code is available at: https://github.com/qxzha/ReCon.

多模态匹配关系一致性图像文本对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。