通过图神经网络融合相似样本语义，自动修正跨模态数据中的错误配对。
Intra-Modal Neighbors Never Lie: Rectifying Inter-Modal Noisy Correspondence via Graph-Based Intra-Modal Reasoning
- 利用动态记忆库中同模态邻居的语义关系进行推理
- 在Flickr30K等数据集上提升跨模态检索准确率超过5%
- 适合需要高精度跨模态匹配的图像-文本应用
大规模网络采集的数据集推动了跨模态检索的发展，但不可避免地包含噪声对应关系，严重损害模型泛化能力。现有方法主要通过过滤噪声或寻找替代标签来应对，但大多受限于'离散选择'范式。我们提出新框架IN2R，从寻找替代标签转向合成可靠监督信号。基于同模态数据的内在几何稳定性，IN2R使用图重构器在动态跨模型记忆库中检索邻居，并通过关系推理生成反映局部语义共识的连续软原型，有效纠正跨模态错配。在Flickr30K、MS-COCO和CC152K上的大量实验表明，IN2R显著优于现有最先进方法。代码与预训练模型已公开于https://github.com/liuyyy111/IN2R。
原文摘要 · Abstract (English)
Large-scale web-harvested datasets have fueled the progress of cross-modal retrieval but inevitably suffer from noisy correspondence, which severely degrades model generalization. Existing methods primarily address this by filtering out noise or seeking a substitute label, yet they predominantly remain bound by a "Discrete Selection" paradigm. We argue that relying on a single discrete proxy induces Single-Point Fragility and Discretization Error. To overcome these limitations, we propose a novel framework, Intra-modal Neighbor-aware Noise Rectification (IN2R), which shifts the paradigm from searching for a substitute to synthesizing a reliable supervision target. Leveraging the intrinsic geometric stability of intra-modal data, IN2R employs a Graph Refiner to perform relational reasoning over neighbors retrieved from a dynamic Cross-Model Memory. Instead of propagating discrete labels, our method synthesizes a continuous, soft prototype that reflects the consensus of the local semantic neighborhood, effectively rectifying inter-modal misalignment. Extensive experiments on Flickr30K, MS-COCO, and CC152K demonstrate that IN2R significantly outperforms state-of-the-art methods. Our code and pre-trained models are publicly available at https://github.com/liuyyy111/IN2R.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。