让视觉检索更懂逻辑,通过显式补全隐含语义提升跨模态搜索能力
Reasoning-Augmented Representations for Multimodal Retrieval
- 先用大模型为图像生成密集描述,把隐藏信息显性化
- 在M-BEIR数据集上,对需要推理的查询任务提升显著
- 适合处理复杂语义、组合修改等高阶检索需求
通用多模态检索(UMR)旨在实现文本与视觉间的任意搜索,但当前嵌入模型在需潜在推理的查询(如解析不明确指代或匹配组合约束)面前仍易失效。我们指出这种脆弱性常源于数据:当图像包含‘无声’证据而查询遗漏关键语义时,单次嵌入需同时完成推理与压缩,易导致错误特征匹配。为此提出以数据为中心的框架,将推理前置并外化:利用强大视觉-语言模型,对语料库条目进行密集图像描述生成,显化视觉证据,解析查询中的模糊指代,并将冗长指令重写为简洁检索约束。仅推理时增强不足;检索器必须在这些语义密集表示上训练,以避免分布偏移并充分挖掘新增信号。在M-BEIR基准上,该方法持续优于强基线,消融实验表明语料增强主要惠及知识密集型查询,而查询增强对组合修改请求至关重要。代码已公开于https://github.com/AugmentedRetrieval/ReasoningAugmentedRetrieval。
原文摘要 · Abstract (English)
Universal Multimodal Retrieval (UMR) seeks any-to-any search across text and vision, yet modern embedding models remain brittle when queries require latent reasoning (e.g., resolving underspecified references or matching compositional constraints). We argue this brittleness is often data-induced: when images carry "silent" evidence and queries leave key semantics implicit, a single embedding pass must both reason and compress, encouraging spurious feature matching. We propose a data-centric framework that decouples these roles by externalizing reasoning before retrieval. Using a strong Vision--Language Model, we make implicit semantics explicit by densely captioning visual evidence in corpus entries, resolving ambiguous multimodal references in queries, and rewriting verbose instructions into concise retrieval constraints. Inference-time enhancement alone is insufficient; the retriever must be trained on these semantically dense representations to avoid distribution shift and fully exploit the added signal. Across M-BEIR, our reasoning-augmented training method yields consistent gains over strong baselines, with ablations showing that corpus enhancement chiefly benefits knowledge-intensive queries while query enhancement is critical for compositional modification requests. We publicly release our code at https://github.com/AugmentedRetrieval/ReasoningAugmentedRetrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。