修复大模型检索时的细粒度推理能力下降问题
ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrieval
- 通过诊断-生成-精炼三阶段恢复检索模型的细粒度理解能力
- 在CIRR和FashionIQ上达到当前最佳性能,显著提升检索准确率
- 适用于需要精准图像组合检索的应用场景
组合图像检索(CIR)旨在根据包含参考图像和修改文本的混合查询检索目标图像。早期双塔视觉语言模型在跨模态组合推理方面表现不佳。尽管将生成式多模态大模型(MLLM)用于检索具有前景,但我们发现该策略忽略了根本问题:将生成式MLLM压缩为单嵌入判别式检索器会引发范式冲突,导致能力退化——即检索适应后原生细粒度推理能力下降。为此,我们提出ReCALL,一种模型无关框架,遵循诊断-生成-精炼流程:首先通过自引导信息实例挖掘诊断检索器的认知盲区;接着利用基础MLLM生成修正指令和三元组,并通过VQA一致性过滤进行质量控制;最后通过分组对比学习在这些三元组上持续训练检索器,从而内化细粒度视觉-语义区分,重新对齐检索器的判别嵌入空间与MLLM内部的固有组合推理能力。在CIRR和FashionIQ上的大量实验表明,ReCALL能持续校准退化的推理能力,达到最先进水平。代码已公开于https://github.com/RemRico/Recall。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) aims to retrieve target images based on a hybrid query comprising a reference image and a modification text. Early dual-tower Vision-Language Models (VLMs) struggle with cross-modality compositional reasoning required for this task. While adapting generative Multimodal Large Language Models (MLLMs) for retrieval offers a promising direction, we identify that this strategy overlooks a fundamental issue: compressing a generative MLLM into a single-embedding discriminative retriever triggers a paradigm conflict, which leads to Capability Degradation - the deterioration of native fine-grained reasoning after retrieval adaptation. To address this challenge, we propose ReCALL, a model-agnostic framework that follows a diagnose-generate-refine pipeline: First, we diagnose cognitive blind spots of the retriever via self-guided informative instance mining. Next, we generate corrective instructions and triplets by prompting the foundation MLLM and conduct quality control with VQA-based consistency filtering. Finally, we refine the retriever through continual training on these triplets with a grouped contrastive scheme, thereby internalizing fine-grained visual-semantic distinctions and realigning the discriminative embedding space of retriever with intrinsic compositional reasoning within the MLLM. Extensive experiments on CIRR and FashionIQ show that ReCALL consistently recalibrates degraded capabilities and achieves state-of-the-art performance. Code is available at https://github.com/RemRico/Recall.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。