构建医学实体树,让大模型更懂临床知识的层级关系。
Learning from Medical Entity Trees: An Entity-Centric Medical Data Engineering Framework for MLLMs

- 以医学实体树为核心,整合疾病、解剖、症状等知识
- 通过节点引导检索与知识约束生成,提升图文匹配精度
- 适合医疗多模态大模型训练,尤其提升复杂问诊能力
多模态大语言模型在医疗应用中展现出巨大潜力,但其性能受限于传统粗粒度按模态或科室划分的数据处理方式。这类碎片化方法无法捕捉临床医学知识的层次性和关联性,制约了模型在细粒度识别与复杂推理方面的能力。本文提出一种面向多模态大模型的实体中心医学数据工程框架。我们从权威医学文献中自动提取实体,构建医学实体树(Medical Entity Tree, MET),将疾病、解剖结构、检查模态和症状系统性地组织为统一知识库。基于该树结构,设计了三项核心技术:(1) 节点引导检索,将原始数据锚定至具体医学概念;(2) 两阶段混合过滤与对齐流程,确保视觉-语义精确对应;(3) 知识感知的数据合成,生成丰富描述与针对性推理型VQA问答对,利用结构约束增强语义一致性。在六个医学基准上的大量实验表明,该方法显著提升通用多模态大模型的医疗能力,在处理复杂临床问题上表现优异,达到多样化医疗场景下的先进水平。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown transformative potential in medical applications, yet their performance is hindered by conventional data curation strategies that rely on coarse-grained partitioning by modality or department. Such fragmented approaches fail to capture the hierarchical and interconnected nature of clinical medical knowledge, limiting the models' ability to perform fine-grained recognition and complex reasoning. In this paper, we propose a novel Entity-Centric Medical Data Engineering framework. We automatically extract entities from authoritative medical literature to construct a Medical Entity Tree (MET), a hierarchical structure that systematically encodes diseases, anatomical structures, modalities, and symptoms into a unified knowledge repository. Building upon the MET, we propose an advanced data engine that includes: (1) node-guided retrieval to anchor raw data to specific medical concepts, (2) a two-stage hybrid filtering and alignment pipeline to ensure precise visual-semantic correspondence, and (3) knowledge-aware data synthesis to generate enriched captions and targeted reasoning VQA pairs, leveraging structural constraints. Extensive evaluations across six medical benchmarks demonstrate that our approach significantly enhances the medical capabilities of general-purpose MLLMs, improving their ability to handle complex clinical queries and achieve state-of-the-art performance in diverse medical contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。