用图像检索提升医学问答准确率,比文本检索高5.8分。
Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering

- 直接检索医学文献页面图像,融合视觉与语言信息进行推理。
- 在4个基准上平均准确率达78.6%,较无检索基线提升5.8分。
- 支持多轮迭代记忆积累,适合医疗领域高精度问答场景。
医学检索增强生成系统通常仅使用从生物医学文献中提取的文本块,忽略了原始文档页中的丰富视觉内容(如表格、图表、结构化布局)。我们提出MED-VRAG,一种迭代式多模态RAG框架,直接对PMC文档页面图像进行检索与推理,而非依赖OCR文本。该系统将ColQwen2.5的像素级页面嵌入与分片式MapReduce LLM过滤器结合,实现约35万页的可扩展性,且第一阶段检索耗时低于30毫秒,得益于离线粗到精索引(每页8个聚类中心,基于聚类的近似最近邻搜索,对前R个候选进行精确双向打分)。随后,视觉-语言模型(VLM)通过最多三轮迭代,不断优化查询并累积证据至记忆库,单轮耗时约15.9秒,三轮总耗时约47.8秒(4xA100)。在四个医学问答基准(MedQA、MedMCQA、PubMedQA、MMLU-Med)上,MED-VRAG达到78.6%的平均准确率。与相同Qwen2.5-VL-32B主干模型相比,检索带来+5.8分增益;相较于MedRAG + GPT-4(76.8%),有+1.8分优势(跨论文比较,非直接对比)。消融实验表明:页面图像检索贡献+1.0,迭代机制贡献+1.5,记忆库贡献+1.0。
原文摘要 · Abstract (English)
Medical retrieval-augmented generation (RAG) systems typically operate on text chunks extracted from biomedical literature, discarding the rich visual content (tables, figures, structured layouts) of original document pages. We propose MED-VRAG, an iterative multimodal RAG framework that retrieves and reasons over PMC document page images instead of OCR'd text. The system pairs ColQwen2.5 patch-level page embeddings with a sharded MapReduce LLM filter, scaling to ~350K pages while keeping Stage-1 retrieval under 30 ms via an offline coarse-to-fine index (C=8 centroids per page, ANN over centroids, exact two-way scoring on the top-R shortlist). A vision-language model (VLM) then iteratively refines its query and accumulates evidence in a memory bank across up to 3 reasoning rounds, with a single iteration costing ~15.9 s and the full three-round pipeline ~47.8 s on 4xA100. Across four medical QA benchmarks (MedQA, MedMCQA, PubMedQA, MMLU-Med), MEDVRAG reaches 78.6% average accuracy. Under controlled comparison with the same Qwen2.5-VL-32B backbone, retrieval contributes a +5.8 point gain over the no-retrieval baseline; we also note a +1.8 point edge over MedRAG + GPT-4 (76.8%), with the caveat that this is a cross-paper rather than head-to-head comparison. Ablations isolate +1.0 from page-image vs text-chunk retrieval, +1.5 from iteration, and +1.0 from the memory bank.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。