arXiv:2412.14535cs.CV2024-12AAAI被引 8

模仿医生写报告的两阶段流程,提升医学影像报告生成质量

DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching

论文配图:DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching
图 1 · 摘自论文原文
  • 分两阶段生成:先粗对齐梅奥主题词,再细建图像与文本的高阶关系
  • 在多个公开数据集上优于现有方法,关键短语准确率显著提升
  • 适合需要高精度医学报告生成的研究者和临床辅助系统开发者

医学报告生成对临床诊断和患者管理至关重要,需基于医学影像总结诊断与建议。然而,现有方法常忽略医生写报告的实际流程——先快速概览,再详细分析。此外,当前对齐方式易导致语义错配。为此,我们提出DAMPER,一种双阶段医学报告生成框架,模拟临床写报告的流程。第一阶段为梅奥主题词引导的粗粒度对齐(MCG),将胸部X光(CXR)图像特征与医学主题词(MeSH)特征对齐,生成整体印象的粗略关键词表示。第二阶段为超图增强的细粒度对齐(HFG),为图像块和报告标注构建超图,建模模态内高阶关系,并通过超图匹配捕捉图像区域与文本短语间的语义关联。最终,将粗粒度视觉特征、生成的MeSH表示和视觉超图特征输入报告解码器,生成最终报告。在多个公开数据集上的实验表明,DAMPER在生成全面且准确的医学报告方面表现优异,各项评估指标均超越当前最优方法。

原文摘要 · Abstract (English)

Medical report generation is crucial for clinical diagnosis and patient management, summarizing diagnoses and recommendations based on medical imaging. However, existing work often overlook the clinical pipeline involved in report writing, where physicians typically conduct an initial quick review followed by a detailed examination. Moreover, current alignment methods may lead to misaligned relationships. To address these issues, we propose DAMPER, a dual-stage framework for medical report generation that mimics the clinical pipeline of report writing in two stages. In the first stage, a MeSH-Guided Coarse-Grained Alignment (MCG) stage that aligns chest X-ray (CXR) image features with medical subject headings (MeSH) features to generate a rough keyphrase representation of the overall impression. In the second stage, a Hypergraph-Enhanced Fine-Grained Alignment (HFG) stage that constructs hypergraphs for image patches and report annotations, modeling high-order relationships within each modality and performing hypergraph matching to capture semantic correlations between image regions and textual phrases. Finally,the coarse-grained visual features, generated MeSH representations, and visual hypergraph features are fed into a report decoder to produce the final medical report. Extensive experiments on public datasets demonstrate the effectiveness of DAMPER in generating comprehensive and accurate medical reports, outperforming state-of-the-art methods across various evaluation metrics.

医学报告生成多模态对齐超图建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。