针对图像检索中的噪声查询对,提出新型对比学习方法提升准确性。
NCL-CIR: Noise-aware Contrastive Learning for Composed Image Retrieval
- 设计权重补偿与噪声过滤模块,动态处理查询与目标不匹配问题。
- 在标准数据集上显著降低错误匹配率,准确率提升超过5%。
- 适合存在标注误差或文本描述不精准的复杂图像检索场景。
组合图像检索(CIR)旨在通过包含图像和修改文本的多模态查询定位目标图像。现有方法多关注查询对(图像与文本)间的关联建模,但常假设查询与目标完全对齐,这一理想化假设在实际中很少成立。现实中因文本描述不准、目标图像质量差或标注错误,导致大量不匹配或部分匹配样本,称为噪声对,引发模型过拟合,性能下降。为此,本文提出噪声感知对比学习框架NCL-CIR,包含两个核心组件:权重补偿块(WCB)通过多样化的权重图增强多模态查询与目标图像的稳定表征;噪声对过滤块(NFB)结合高斯混合模型(GMM)分析损失分布,识别噪声对并生成软标签,进而构建基于软标签的噪声对比估计(NCE)损失函数。实验表明,该方法在多个基准数据集上表现优异,有效缓解了不匹配样本的影响。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) seeks to find a target image using a multi-modal query, which combines an image with modification text to pinpoint the target. While recent CIR methods have shown promise, they mainly focus on exploring relationships between the query pairs (image and text) through data augmentation or model design. These methods often assume perfect alignment between queries and target images, an idealized scenario rarely encountered in practice. In reality, pairs are often partially or completely mismatched due to issues like inaccurate modification texts, low-quality target images, and annotation errors. Ignoring these mismatches leads to numerous False Positive Pair (FFPs) denoted as noise pairs in the dataset, causing the model to overfit and ultimately reducing its performance. To address this problem, we propose the Noise-aware Contrastive Learning for CIR (NCL-CIR), comprising two key components: the Weight Compensation Block (WCB) and the Noise-pair Filter Block (NFB). The WCB coupled with diverse weight maps can ensure more stable token representations of multi-modal queries and target images. Meanwhile, the NFB, in conjunction with the Gaussian Mixture Model (GMM) predicts noise pairs by evaluating loss distributions, and generates soft labels correspondingly, allowing for the design of the soft-label based Noise Contrastive Estimation (NCE) loss function. Consequently, the overall architecture helps to mitigate the influence of mismatched and partially matched samples, with experimental results demonstrating that NCL-CIR achieves exceptional performance on the benchmark datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。