MELT提升图像修改检索，解决罕见语义忽略与噪声干扰问题
MELT: Improve Composed Image Retrieval via the Modification Frequentation-Rarity Balance Network
- 通过频率-稀有度平衡机制增强对罕见修改语义的关注
- 利用扩散去噪处理高相似度难负样本，提升匹配鲁棒性
- 在两个基准数据集上显著优于现有方法，适合图像编辑场景
组合图像检索（CIR）以参考图像和修改文本作为查询，旨在检索满足“根据文本指令修改参考图像”要求的目标图像。然而，现有方法存在两大局限：（1）频率偏差导致“稀有样本被忽略”，（2）相似度分数易受难负样本和噪声干扰。为此，我们面对两个核心挑战：非对称稀有语义定位与难负样本下的鲁棒相似度估计。为此，我们提出修改频率-稀有度平衡网络MELT。MELT在多模态上下文中增加对稀有修改语义的关注，同时对高相似度的难负样本应用基于扩散的去噪，从而增强多模态融合与匹配效果。在两个CIR基准上的大量实验验证了MELT的优越性能。代码已开源于https://github.com/luckylittlezhi/MELT。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) uses a reference image and a modification text as a query to retrieve a target image satisfying the requirement of ``modifying the reference image according to the text instructions''. However, existing CIR methods face two limitations: (1) frequency bias leading to ``Rare Sample Neglect'', and (2) susceptibility of similarity scores to interference from hard negative samples and noise. To address these limitations, we confront two key challenges: asymmetric rare semantic localization and robust similarity estimation under hard negative samples. To solve these challenges, we propose the Modification frEquentation-rarity baLance neTwork MELT. MELT assigns increased attention to rare modification semantics in multimodal contexts while applying diffusion-based denoising to hard negative samples with high similarity scores, enhancing multimodal fusion and matching. Extensive experiments on two CIR benchmarks validate the superior performance of MELT. Codes are available at https://github.com/luckylittlezhi/MELT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。