用知识图谱增强医学影像解释,提升准确性和隐私保护。
LLaVA Needs More Knowledge: Retrieval Augmented Natural Language Generation with Knowledge Graph for Explaining Thoracic Pathologies
- 引入知识图谱检索机制,补充模型医疗领域知识
- 在MIMIC-NLE数据集上达顶尖性能,生成更精准解释
- 适合医学AI、临床辅助诊断方向的研究者参考
为医学图像(尤其是胸部病灶)的模型预测生成自然语言解释(NLEs),仍是关键且具挑战性的任务。现有方法常因通用模型缺乏专业医学知识,以及基于检索的增强技术引发隐私问题而受限。为此,我们提出一种基于知识图谱(KG)的数据存储增强的视觉-语言框架,通过引入领域专属医学知识,提升模型对胸部病灶解释的准确性与信息量。该框架采用基于知识图谱的检索机制,在提高解释精度的同时,避免直接数据检索,有效保护隐私。知识图谱模块可即插即用,兼容多种模型架构。我们构建并评估了三种框架:KG-LLaVA(融合预训练LLaVA与KG-RAG)、Med-XPT(结合MedCLIP、Transformer投影器与GPT-2)、Bio-LLaVA(以Bio-ViT-L替换视觉主干)。在MIMIC-NLE数据集上的实验表明,这些框架均达到当前最优表现,验证了知识图谱增强在生成高质量胸部病灶解释中的有效性。
原文摘要 · Abstract (English)
Generating Natural Language Explanations (NLEs) for model predictions on medical images, particularly those depicting thoracic pathologies, remains a critical and challenging task. Existing methodologies often struggle due to general models' insufficient domain-specific medical knowledge and privacy concerns associated with retrieval-based augmentation techniques. To address these issues, we propose a novel Vision-Language framework augmented with a Knowledge Graph (KG)-based datastore, which enhances the model's understanding by incorporating additional domain-specific medical knowledge essential for generating accurate and informative NLEs. Our framework employs a KG-based retrieval mechanism that not only improves the precision of the generated explanations but also preserves data privacy by avoiding direct data retrieval. The KG datastore is designed as a plug-and-play module, allowing for seamless integration with various model architectures. We introduce and evaluate three distinct frameworks within this paradigm: KG-LLaVA, which integrates the pre-trained LLaVA model with KG-RAG; Med-XPT, a custom framework combining MedCLIP, a transformer-based projector, and GPT-2; and Bio-LLaVA, which adapts LLaVA by incorporating the Bio-ViT-L vision model. These frameworks are validated on the MIMIC-NLE dataset, where they achieve state-of-the-art results, underscoring the effectiveness of KG augmentation in generating high-quality NLEs for thoracic pathologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。