MIRA提升医学多模态模型的准确率,解决检索过多或过少的问题。
MIRA: A Novel Framework for Fusing Modalities in Medical RAG
- 动态调整检索数量,避免信息遗漏或干扰
- 融合图像与医学知识库,提升推理准确性
- 适合医疗AI研发者和临床辅助系统开发者
多模态大语言模型在辅助医疗诊断中取得进展,但常产生与医学常识不符的回答。检索增强生成(RAG)虽能提升事实准确性,却面临两大挑战:检索不足会遗漏关键信息,检索过多则引入无关或误导内容;即使初始回答正确,过度依赖检索数据仍可能导致事实错误。为此,我们提出多模态智能检索与增强框架(MIRA),包含两个核心组件:(1)校准的重思与重排模块,动态调节检索上下文数量以控制事实风险;(2)整合图像嵌入与医学知识库的RAG框架,结合查询重写模块实现高效多模态推理。该框架使模型能有效融合自身知识与外部参考。在公开医疗视觉问答与报告生成基准上的评估显示,MIRA显著提升事实准确性与整体性能,达到新最佳水平。代码已开源。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have significantly advanced AI-assisted medical diagnosis, but they often generate factually inconsistent responses that deviate from established medical knowledge. Retrieval-Augmented Generation (RAG) enhances factual accuracy by integrating external sources, but it presents two key challenges. First, insufficient retrieval can miss critical information, whereas excessive retrieval can introduce irrelevant or misleading content, disrupting model output. Second, even when the model initially provides correct answers, over-reliance on retrieved data can lead to factual errors. To address these issues, we introduce the Multimodal Intelligent Retrieval and Augmentation (MIRA) framework, designed to optimize factual accuracy in MLLM. MIRA consists of two key components: (1) a calibrated Rethinking and Rearrangement module that dynamically adjusts the number of retrieved contexts to manage factual risk, and (2) A medical RAG framework integrating image embeddings and a medical knowledge base with a query-rewrite module for efficient multimodal reasoning. This enables the model to effectively integrate both its inherent knowledge and external references. Our evaluation of publicly available medical VQA and report generation benchmarks demonstrates that MIRA substantially enhances factual accuracy and overall performance, achieving new state-of-the-art results. Code is released at https://github.com/mbzuai-oryx/MIRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。