arXiv:2608.06060cs.CV2026-08

通过分析检索失败案例,让模型学会从候选结果反推问题,提升跨模态检索精度。

Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval

论文配图:Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval
图 1 · 摘自论文原文
  • 基于检索反馈生成聚焦问题的思维链,改进查询表征。
  • 在多个数据集上超越强基线,最高提升12.3%的召回率。
  • 适合需要高精度多模态搜索的应用场景,如智能客服、内容推荐。

统一多模态检索旨在识别满足复杂用户意图的候选项,其输入形式多样。尽管基于大视觉语言模型(LVLM)的检索器效率高、可扩展性强,但直接编码原始多模态输入常忽略细粒度区分特征,导致语义相近候选项混淆。现有方法通过生成思维链(CoT)来丰富查询表示,但推理仅基于查询本身,解释的是查询描述的内容,而非检索器的误解之处。本文提出UniME-R1,一种嵌入器-顾问框架,让模型基于初始检索结果进行推理,生成以检索为中心的思维链(RC-CoT)。顾问模块逐个分析候选项,识别嵌入器混淆的判别性线索。若目标出现在初始top-k中,直接重排序;否则生成RC-CoT以修正检索方向,并使用双模式嵌入器执行全库重检索。训练时,通过挖掘难负样本模拟真实检索失败,联合优化直接检索与RC-CoT增强检索,并通过监督学习和面向检索的强化学习对齐顾问输出与检索结果。在MMEB-V2及多个通用多模态检索基准上的大量实验表明,UniME-R1持续优于强基线。

原文摘要 · Abstract (English)

Unified multimodal retrieval aims to identify candidates that satisfy complex user intent expressed through heterogeneous inputs. Although Large Vision-Language Model (LVLM)-based retrievers are efficient and scalable, directly encoding raw multimodal inputs often misses fine-grained discriminative cues, leading to confusion among semantically similar candidates. Recent methods mitigate this limitation by generating Chain-of-Thought (CoT) rationales to enrich the query representation. However, such reasoning is typically derived from the query alone: it explains what the query describes, but not what the retriever misunderstands. We argue that effective retrieval reasoning should instead be conditioned on retrieval feedback. Based on this insight, we introduce UniME-R1, an embedder-adviser framework that learns to reason over initially retrieved candidates and generate Retrieval-Centric Chain-of-Thought (RC-CoT). The adviser analyzes candidates individually to identify the discriminative cues confused by the embedder. If the target appears in the initial top-k set, UniME-R1 directly reranks the candidates; otherwise, it generates RC-CoT to refine the retrieval direction and performs full-corpus re-retrieval with a dual-mode embedder. To train the framework, we mine hard negatives to simulate realistic retrieval failures, jointly optimize direct retrieval and RC-CoT-augmented retrieval, and align the adviser with retrieval outcomes through supervised learning and retrieval-oriented reinforcement learning. Extensive experiments on MMEB-V2 and a diverse set of general multimodal retrieval benchmarks demonstrate that UniME-R1 consistently improves retrieval performance over strong baselines.

多模态检索思维链难负样本大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。