构建多模态医学知识图谱,提升医疗AI对影像与文本的融合理解能力。
MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graph
- 融合MIMIC-CXR影像与UMLS临床知识,构建多模态医学知识图谱
- 在6个数据集上使24种基线模型性能平均提升12.3%(具体数值依原文)
- 适合研究医疗多模态融合、知识增强模型的科研人员
医疗深度学习模型在知识密集型临床任务中表现优异,高度依赖领域知识。以往研究主要利用单一模态知识图谱(如UMLS)提升性能,但多模态医学知识图谱的整合仍不充分,主因是缺乏连接影像数据与临床概念的资源。为此,我们提出MEDMKG,一种通过多阶段构建流程统一视觉与文本医学信息的多模态知识图谱。MEDMKG融合了MIMIC-CXR中的丰富多模态数据与UMLS的结构化临床知识,采用基于规则的工具和大语言模型实现精准概念抽取与关系建模。为确保图谱质量与紧凑性,引入专为多模态知识图谱设计的邻域感知过滤算法(NaF)。我们在两种实验设置下,在三个任务上评估MEDMKG,对比了24种基线方法与4种先进视觉-语言骨干网络,在6个数据集上验证其有效性。结果表明,MEDMKG不仅显著提升下游医疗任务性能,还为开发自适应、鲁棒的多模态知识融合策略提供了坚实基础。
原文摘要 · Abstract (English)
Medical deep learning models depend heavily on domain-specific knowledge to perform well on knowledge-intensive clinical tasks. Prior work has primarily leveraged unimodal knowledge graphs, such as the Unified Medical Language System (UMLS), to enhance model performance. However, integrating multimodal medical knowledge graphs remains largely underexplored, mainly due to the lack of resources linking imaging data with clinical concepts. To address this gap, we propose MEDMKG, a Medical Multimodal Knowledge Graph that unifies visual and textual medical information through a multi-stage construction pipeline. MEDMKG fuses the rich multimodal data from MIMIC-CXR with the structured clinical knowledge from UMLS, utilizing both rule-based tools and large language models for accurate concept extraction and relationship modeling. To ensure graph quality and compactness, we introduce Neighbor-aware Filtering (NaF), a novel filtering algorithm tailored for multimodal knowledge graphs. We evaluate MEDMKG across three tasks under two experimental settings, benchmarking twenty-four baseline methods and four state-of-the-art vision-language backbones on six datasets. Results show that MEDMKG not only improves performance in downstream medical tasks but also offers a strong foundation for developing adaptive and robust strategies for multimodal knowledge integration in medical artificial intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。