arXiv:2412.11070cs.CV2024-12中稿 · AAAI被引 17

用历史影像与报告约束大模型,提升放射科报告生成的连续性。

HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation

  • 通过时序共享与特异性特征捕捉疾病进展
  • 在Longitudinal-MIMIC数据集上达到当前最佳效果
  • 无需历史数据也能生成准确报告,适配多种大模型

放射科报告生成(RRG)模型通常仅关注单次检查,忽略历史影像或文本数据的整合,而这些对患者随访至关重要。传统方法在处理长序列依赖时表现不佳,但大语言模型(LLM)擅长上下文学习,适合分析纵向医疗数据。为此,我们提出一种新型历史约束大语言模型(HC-LLM)框架,通过约束纵向图像与报告间的一致性与差异性,赋予LLM生成连续报告的能力。具体而言,从纵向胸部X光片和诊断报告中提取时间共享与时间特异性特征,以捕捉疾病演变。随后,通过模态内相似性约束和跨模态对比与结构约束,确保特征表示一致性。这些联合约束有效引导LLM生成准确反映疾病进展的诊断报告,在Longitudinal-MIMIC数据集上取得当前最优性能。值得注意的是,该方法在测试时无需历史数据仍表现良好,且可轻松适配其他多模态大模型,具备强通用性。

原文摘要 · Abstract (English)

Radiology report generation (RRG) models typically focus on individual exams, often overlooking the integration of historical visual or textual data, which is crucial for patient follow-ups. Traditional methods usually struggle with long sequence dependencies when incorporating historical information, but large language models (LLMs) excel at in-context learning, making them well-suited for analyzing longitudinal medical data. In light of this, we propose a novel Historical-Constrained Large Language Models (HC-LLM) framework for RRG, empowering LLMs with longitudinal report generation capabilities by constraining the consistency and differences between longitudinal images and their corresponding reports. Specifically, our approach extracts both time-shared and time-specific features from longitudinal chest X-rays and diagnostic reports to capture disease progression. Then, we ensure consistent representation by applying intra-modality similarity constraints and aligning various features across modalities with multimodal contrastive and structural constraints. These combined constraints effectively guide the LLMs in generating diagnostic reports that accurately reflect the progression of the disease, achieving state-of-the-art results on the Longitudinal-MIMIC dataset. Notably, our approach performs well even without historical data during testing and can be easily adapted to other multimodal large models, enhancing its versatility.

医学报告生成多模态大模型纵向数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。