通过细粒度概率建模,提升图文组合检索的鲁棒性
Heterogeneous Uncertainty-Guided Composed Image Retrieval with Fine-Grained Probabilistic Learning
- 用高斯嵌入分别表示查询与目标,捕捉细节概念与不确定性
- 针对多模态查询与单模态目标设计异构不确定性估计,动态加权
- 引入不确定性引导的对比学习,显著提升检索精度
组合图像检索(CIR)通过参考图像与修改文本实现图像搜索。然而,CIR三元组中的固有噪声带来了内在不确定性,威胁模型鲁棒性。尽管概率学习方法展现出潜力,但其在实例级别进行整体建模且对查询与目标一视同仁,难以适配CIR。本文提出异构不确定性引导(HUG)范式,采用细粒度概率学习框架,将查询与目标表示为高斯嵌入,以捕捉详细概念与不确定性。针对多模态查询和单模态目标,定制化设计异构不确定性估计。给定查询时,不仅评估单模态内容质量的不确定性,还捕捉多模态协调性不确定性,并通过可证明的动态加权机制生成综合查询不确定性。进一步设计不确定性引导的目标函数,包括查询-目标整体对比和细粒度对比,结合全面的负样本策略,有效增强判别性学习。在多个基准测试上,HUG表现超越现有最先进方法,实验分析验证了技术贡献的有效性。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) enables image search by combining a reference image with modification text. Intrinsic noise in CIR triplets incurs intrinsic uncertainty and threatens the model's robustness. Probabilistic learning approaches have shown promise in addressing such issues; however, they fall short for CIR due to their instance-level holistic modeling and homogeneous treatment of queries and targets. This paper introduces a Heterogeneous Uncertainty-Guided (HUG) paradigm to overcome these limitations. HUG utilizes a fine-grained probabilistic learning framework, where queries and targets are represented by Gaussian embeddings that capture detailed concepts and uncertainties. We customize heterogeneous uncertainty estimations for multi-modal queries and uni-modal targets. Given a query, we capture uncertainties not only regarding uni-modal content quality but also multi-modal coordination, followed by a provable dynamic weighting mechanism to derive comprehensive query uncertainty. We further design uncertainty-guided objectives, including query-target holistic contrast and fine-grained contrasts with comprehensive negative sampling strategies, which effectively enhance discriminative learning. Experiments on benchmarks demonstrate HUG's effectiveness beyond state-of-the-art baselines, with faithful analysis justifying the technical contributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。