用历史报告引导双模态学习,提升病理报告生成质量。
Historical Report Guided Bi-modal Concurrent Learning for Pathology Report Generation
- 通过检索医学知识库补充视觉特征语义信息。
- 双模态并行学习使报告生成准确率提升19.1%。
- 适合医学AI研究者与临床辅助诊断开发人员。
从全切片图像(WSIs)自动生成病理报告面临两大挑战:视觉特征语义贫乏和WSIs固有的信息冗余。为此,我们提出一种模拟病理科医生诊断推理的新型历史报告引导双模态并发学习框架(BiGen),包含:(1) 知识检索机制,通过匹配高注意力区域块从预建医学知识库中检索相关知识以丰富语义;(2) 双模态并发学习策略,利用可学习的视觉标记和文本标记动态提取关键视觉特征与检索知识,权重共享层实现视觉与知识特征的跨模态对齐。多模态解码器融合双模态信息生成综合诊断报告。在PathText(BRCA)数据集上的实验表明,该框架表现优异,相比现有方法在NLP指标上相对提升7.4%,在Her-2预测分类指标上提升19.1%。消融实验证明各模块必要性,凸显其提供相关语义内容并抑制冗余信息的能力。代码已公开于https://github.com/DeepMed-Lab-ECNU/BiGen。
原文摘要 · Abstract (English)
Automated pathology report generation from Whole Slide Images (WSIs) faces two key challenges: (1) lack of semantic content in visual features and (2) inherent information redundancy in WSIs. To address these issues, we propose a novel Historical Report Guided \textbf{Bi}-modal Concurrent Learning Framework for Pathology Report \textbf{Gen}eration (BiGen) emulating pathologists' diagnostic reasoning, consisting of: (1) A knowledge retrieval mechanism to provide rich semantic content, which retrieves WSI-relevant knowledge from pre-built medical knowledge bank by matching high-attention patches and (2) A bi-modal concurrent learning strategy instantiated via a learnable visual token and a learnable textual token to dynamically extract key visual features and retrieved knowledge, where weight-shared layers enable cross-modal alignment between visual features and knowledge features. Our multi-modal decoder integrates both modals for comprehensive diagnostic reports generation. Experiments on the PathText (BRCA) dataset demonstrate our framework's superiority, achieving state-of-the-art performance with 7.4\% relative improvement in NLP metrics and 19.1\% enhancement in classification metrics for Her-2 prediction versus existing methods. Ablation studies validate the necessity of our proposed modules, highlighting our method's ability to provide WSI-relevant rich semantic content and suppress information redundancy in WSIs. Code is publicly available at https://github.com/DeepMed-Lab-ECNU/BiGen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。