让多模态大模型动态选择检索专家,更精准获取外部知识。
Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitation
- 根据推理状态动态选择最合适的检索专家。
- 在多个基准上平均提升超7%,显著减少幻觉。
- 适合需要高可靠性的多模态问答场景。
多模态检索增强生成(MRAG)通过引入外部知识,在缓解多模态大语言模型(MLLM)幻觉方面展现出潜力。然而,现有方法通常遵循固定的检索范式,模仿预设的检索路径,难以根据模型的知识需求或推理状态动态协调不同检索专家。为此,本文提出混合检索专家(Mixture-of-Retrieval Experts, MoRE),使MLLM能与多样化的检索专家协作,更高效地利用知识。MoRE通过学习动态决定当前应调用哪个专家,其决策基于不断演化的推理状态。为有效训练该能力,我们提出分步组相对策略优化(Stepwise Group Relative Policy Optimization, Step-GRPO),超越稀疏结果反馈,鼓励模型与多个检索专家交互并整合细粒度奖励,从而学会在回答问题时充分协调所有专家。在多个开放域问答基准上的实验表明,MoRE相比主流基线平均性能提升超过7%。值得注意的是,MoRE能动态协调异构专家,精准定位相关知识,验证了其在推理驱动下的鲁棒专家协作能力。所有代码与数据已开源至 https://github.com/OpenBMB/MoRE。
原文摘要 · Abstract (English)
Multimodal Retrieval-Augmented Generation (MRAG) has shown promise in mitigating hallucinations in Multimodal Large Language Models (MLLMs) by incorporating external knowledge. However, existing methods typically adhere to rigid retrieval paradigms by mimicking fixed retrieval trajectories and thus fail to fully exploit the knowledge of different retrieval experts through dynamic interaction based on the model's knowledge needs or evolving reasoning states. To overcome this limitation, we introduce Mixture-of-Retrieval Experts (MoRE), a novel framework that enables MLLMs to collaboratively interact with diverse retrieval experts for more effective knowledge exploitation. Specifically, MoRE learns to dynamically determine which expert to engage with, conditioned on the evolving reasoning state. To effectively train this capability, we propose Stepwise Group Relative Policy Optimization (Step-GRPO), which goes beyond sparse outcome-based supervision by encouraging MLLMs to interact with multiple retrieval experts and synthesize fine-grained rewards, thereby teaching the MLLM to fully coordinate all experts when answering a given query. Experimental results on diverse open-domain QA benchmarks demonstrate the effectiveness of MoRE, achieving average performance gains of over 7% compared to competitive baselines. Notably, MoRE exhibits strong adaptability by dynamically coordinating heterogeneous experts to precisely locate relevant information, validating its capability for robust, reasoning-driven expert collaboration. All codes and data are released on https://github.com/OpenBMB/MoRE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。