用视觉检索增强生成，减少医学多模态大模型的幻觉。
Reducing Hallucinations of Medical Multimodal Large Language Models with Visual Retrieval-Augmented Generation
- 引入视觉检索增强生成框架，结合图像与文本信息
- 在常见和罕见医学实体上均提升识别准确率
- 适合需要高精度医疗报告生成的研究与临床应用
多模态大语言模型在视觉与文本任务中表现优异，但在医疗等关键领域仍存在幻觉问题。本文提出视觉检索增强生成（V-RAG）框架，融合检索图像中的视觉与文本数据。在MIMIC-CXR胸部X光报告生成和Multicare医学图像描述数据集上，V-RAG显著提升了实体探测准确率，尤其对训练数据较少的罕见实体也有改善。进一步应用于幻觉纠正与报告生成，使放射学图谱F1分数更高，提升临床准确性。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown impressive performance in vision and text tasks. However, hallucination remains a major challenge, especially in fields like healthcare where details are critical. In this work, we show how MLLMs may be enhanced to support Visual RAG (V-RAG), a retrieval-augmented generation framework that incorporates both text and visual data from retrieved images. On the MIMIC-CXR chest X-ray report generation and Multicare medical image caption generation datasets, we show that Visual RAG improves the accuracy of entity probing, which asks whether a medical entities is grounded by an image. We show that the improvements extend both to frequent and rare entities, the latter of which may have less positive training data. Downstream, we apply V-RAG with entity probing to correct hallucinations and generate more clinically accurate X-ray reports, obtaining a higher RadGraph-F1 score.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。