让医疗多模态模型主动查资料，提升诊断准确性与可信度。
Proactive Reasoning-with-Retrieval Framework for Medical Multimodal Large Language Models
- 通过两阶段强化学习，让模型结合图像和文本主动检索外部医学知识。
- 在未知领域测试中提升8.8%性能，显著减少幻觉和错误推理。
- 适合需要高可靠性医疗诊断的临床场景，尤其面对罕见病例时。
提升多模态大语言模型（MLLMs）的推理能力对医疗应用至关重要，以实现对医学影像的透明分析并提供可靠诊断。然而，现有医疗MLLMs仅依赖内部知识进行推理，在超出训练范围的病例中容易产生幻觉和事实错误。尽管近期的智能体检索增强生成（RAG）方法能激发模型的主动检索能力，但仅适用于单模态大模型，忽视了推理过程中的关键视觉信息。为此，我们提出首个面向医疗领域的多模态推理-检索框架Med-RwR，可在推理过程中主动查询观察到的症状或领域特定医学概念以检索外部知识。具体而言，设计了两阶段强化学习策略，结合定制奖励机制，促使模型有效利用视觉诊断发现与文本临床信息进行检索。在此基础上，进一步提出置信度驱动的图像重检索（CDIR）方法，在预测置信度低时进行测试时扩展检索。在多个公开医疗基准上的评估表明，Med-RwR显著优于基线模型，证明了引入外部知识对增强推理能力的有效性。此外，该框架在陌生领域展现出卓越泛化能力，在我们提出的超声心动图基准（ECBench）上取得8.8%的性能提升，尽管训练数据中缺乏大量超声心动图样本。相关数据、模型及代码将公开发布于https://github.com/xmed-lab/Med-RwR。
原文摘要 · Abstract (English)
Incentivizing the reasoning ability of Multimodal Large Language Models (MLLMs) is essential for medical applications to transparently analyze medical scans and provide reliable diagnosis. However, existing medical MLLMs rely solely on internal knowledge during reasoning, leading to hallucinated reasoning and factual inaccuracies when encountering cases beyond their training scope. Although recent Agentic Retrieval-Augmented Generation (RAG) methods elicit the medical model's proactive retrieval ability during reasoning, they are confined to unimodal LLMs, neglecting the crucial visual information during reasoning and retrieval. Consequently, we propose the first Multimodal Medical Reasoning-with-Retrieval framework, Med-RwR, which actively retrieves external knowledge by querying observed symptoms or domain-specific medical concepts during reasoning. Specifically, we design a two-stage reinforcement learning strategy with tailored rewards that stimulate the model to leverage both visual diagnostic findings and textual clinical information for effective retrieval. Building on this foundation, we further propose a Confidence-Driven Image Re-retrieval (CDIR) method for test-time scaling when low prediction confidence is detected. Evaluation on various public medical benchmarks demonstrates Med-RwR's significant improvements over baseline models, proving the effectiveness of enhancing reasoning capabilities with external knowledge integration. Furthermore, Med-RwR demonstrates remarkable generalizability to unfamiliar domains, evidenced by 8.8% performance gain on our proposed EchoCardiography Benchmark (ECBench), despite the scarcity of echocardiography data in the training corpus. Our data, model, and codes will be made publicly available at https://github.com/xmed-lab/Med-RwR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。