用大模型主动推理图像与文本关系，显著提升复杂查询的检索效果。
HIVE: Query, Hypothesize, Verify An LLM Framework for Multimodal Reasoning-Intensive Retrieval
- 通过大模型生成视觉-文本推理假设，指导精准检索
- 在MM-BRIGHT上达41.7 nDCG@10，比最优基线高8.5点
- 特别擅长游戏、化学等视觉密集型领域，适合多模态推理任务
多模态检索模型在需深度图文融合的推理型查询中表现不佳，最佳模型在MM-BRIGHT上的nDCG@10仅为27.6，甚至低于强文本检索器（32.2）。我们提出HIVE（假设驱动的迭代视觉证据检索）框架，通过大模型显式注入视觉-文本推理能力。HIVE分四步：（1）初始检索；（2）基于前k个候选的视觉与逻辑缺失，由大模型生成补偿性查询；（3）使用优化后查询进行二次检索；（4）对合并结果进行大模型验证与重排序。在包含2,803个真实世界查询的MM-BRIGHT多模态到文本赛道上，HIVE实现41.7的聚合nDCG@10，较最优文本模型DiVeR（32.2）提升9.5点，较最优多模态模型Nomic-Vision（27.6）提升14.1点。其中，基础检索器贡献33.2，框架本身额外提升8.5点，在游戏（68.2）、化学（42.5）、可持续性（49.4）等视觉密集型领域表现尤为突出。该框架兼容标准与增强型检索器，证明大模型驱动的视觉假设生成与验证可有效弥合多模态检索中的推理差距。
原文摘要 · Abstract (English)
Multimodal retrieval models fail on reasoning-intensive queries where images (diagrams, charts, screenshots) must be deeply integrated with text to identify relevant documents -- the best multimodal model achieves only 27.6 nDCG@10 on MM-BRIGHT, underperforming even strong text-only retrievers (32.2). We introduce \textbf{HIVE} (\textbf{H}ypothesis-driven \textbf{I}terative \textbf{V}isual \textbf{E}vidence Retrieval), a plug-and-play framework that injects explicit visual-text reasoning into a retriever via LLMs. HIVE operates in four stages: (1) initial retrieval over the corpus, (2) LLM-based compensatory query synthesis that explicitly articulates visual and logical gaps observed in top-$k$ candidates, (3) secondary retrieval with the refined query, and (4) LLM verification and reranking over the union of candidates. Evaluated on the multimodal-to-text track of MM-BRIGHT (2,803 real-world queries across 29 technical domains), HIVE achieves a new state-of-the-art aggregated nDCG@10 of \textbf{41.7} -- a \textbf{+9.5} point gain over the best text-only model (DiVeR: 32.2) and \textbf{+14.1} over the best multimodal model (Nomic-Vision: 27.6), where our reasoning-enhanced base retriever contributes 33.2 and the HIVE framework adds a further \textbf{+8.5} points -- with particularly strong results in visually demanding domains (Gaming: 68.2, Chemistry: 42.5, Sustainability: 49.4). Compatible with both standard and reasoning-enhanced retrievers, HIVE demonstrates that LLM-mediated visual hypothesis generation and verification can substantially close the multimodal reasoning gap in retrieval. https://github.com/mm-bright/multimodal-reasoning-retrieval
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。