用知识图谱增强大模型，让放射科报告更精准。
KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Models
- 用知识图谱提取疾病特征，与图像特征融合
- 在MIMIC-CXR和IU-Xray上提升报告质量
- 适合医疗AI、放射科报告自动化场景
利用大语言模型（LLMs）在叙事生成、逻辑推理和常识知识整合方面的强大能力，本研究探索其在自动化放射科报告生成（R2Gen）中的应用。尽管LLMs蕴含丰富知识，但如何高效触发其中与特定任务相关的知识仍是关键挑战。本文提出KARGEN框架，基于冻结的LLM生成报告，并通过知识图谱激活与胸部疾病相关知识，以提升报告的临床价值。通过设计方式从知识图谱中提炼疾病相关特征，再将其与区域图像特征融合，兼顾正常与异常发现。我们探索了两种融合方法，自动优先选择最相关特征，使生成报告对疾病更敏感、质量更高。在MIMIC-CXR和IU-Xray数据集上表现优异。
原文摘要 · Abstract (English)
Harnessing the robust capabilities of Large Language Models (LLMs) for narrative generation, logical reasoning, and common-sense knowledge integration, this study delves into utilizing LLMs to enhance automated radiology report generation (R2Gen). Despite the wealth of knowledge within LLMs, efficiently triggering relevant knowledge within these large models for specific tasks like R2Gen poses a critical research challenge. This paper presents KARGEN, a Knowledge-enhanced Automated radiology Report GENeration framework based on LLMs. Utilizing a frozen LLM to generate reports, the framework integrates a knowledge graph to unlock chest disease-related knowledge within the LLM to enhance the clinical utility of generated reports. This is achieved by leveraging the knowledge graph to distill disease-related features in a designed way. Since a radiology report encompasses both normal and disease-related findings, the extracted graph-enhanced disease-related features are integrated with regional image features, attending to both aspects. We explore two fusion methods to automatically prioritize and select the most relevant features. The fused features are employed by LLM to generate reports that are more sensitive to diseases and of improved quality. Our approach demonstrates promising results on the MIMIC-CXR and IU-Xray datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。