通过地理混合法增强图像文本隐含语义,提升零样本组合检索的多样性与准确率。
G-MIXER: Geodesic Mixup-based Implicit Semantic Expansion and Explicit Semantic Re-ranking for Zero-Shot Composed Image Retrieval

- 利用地理混合法生成多样的图像文本组合特征,捕捉隐含语义
- 在多个基准上实现当前最佳性能,检索准确率显著提升
- 无需训练,适合需要高效部署的零样本检索场景
组合图像检索(CIR)旨在通过参考图像与修改文本的结合来检索目标图像。这要求同时考虑查询中的显性语义和其跨模态组合所蕴含的隐性语义。近期无训练的零样本CIR(ZS-CIR)方法借助多模态大语言模型(MLLMs)生成详细的目标描述,将隐性信息转化为显性文本表达。然而,这些方法过度依赖文本模态,未能充分捕捉需要多样化候选组合的模糊检索特性,导致检索结果多样性与准确性下降。为此,本文提出一种新的无训练方法——基于地理混合法的隐性语义扩展与显性语义重排序(G-MIXER)。G-MIXER通过在多种混合法比例下对参考图像-文本对进行地理混合法,构建反映隐性语义的组合查询特征,并生成多样化候选集;随后利用来自MLLMs的显性语义对候选集进行重排序,从而同时提升检索多样性与准确性。G-MIXER在多个ZS-CIR基准上达到最先进性能,无需额外训练即可有效处理显性与隐性语义。代码将于https://github.com/maya0395/gmixer发布。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) aims to retrieve target images by integrating a reference image with a corresponding modification text. CIR requires jointly considering the explicit semantics specified in the query and the implicit semantics embedded within its bi-modal composition. Recent training-free Zero-Shot CIR (ZS-CIR) methods leverage Multimodal Large Language Models (MLLMs) to generate detailed target descriptions, converting the implicit information into explicit textual expressions. However, these methods rely heavily on the textual modality and fail to capture the fuzzy retrieval nature that requires considering diverse combinations of candidates. This leads to reduced diversity and accuracy in retrieval results. To address this limitation, we propose a novel training-free method, Geodesic Mixup-based Implicit semantic eXpansion and Explicit semantic Re-ranking for ZS-CIR (G-MIXER). G-MIXER constructs composed query features that reflect the implicit semantics of reference image-text pairs through geodesic mixup over a range of mixup ratios, and builds a diverse candidate set. The generated candidates are then re-ranked using explicit semantics derived from MLLMs, improving both retrieval diversity and accuracy. Our proposed G-MIXER achieves state-of-the-art performance across multiple ZS-CIR benchmarks, effectively handling both implicit and explicit semantics without additional training. Our code will be available at https://github.com/maya0395/gmixer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。