用视觉语言嵌入实现病理图像零样本分类,不需标注数据也能精准诊断。
Leveraging Vision-Language Embeddings for Zero-Shot Learning in Histopathology Images
- 多分辨率提取图像块模拟病理医生诊断流程,兼顾细胞细节与组织结构。
- 融合全局与加权局部嵌入,提升对关键病变区域的识别能力。
- 无需微调预训练模型,适合缺乏标注数据的医学图像分析场景。
零样本学习在病理图像分析中具有巨大潜力,可使模型在无大量标注数据的情况下泛化到未见类别。尽管视觉语言模型(VLM)的发展提升了零样本学习能力,但将其应用于病理图像仍面临挑战,主要源于图像复杂性和诊断任务的细微性。本文提出一种名为多分辨率提示引导混合嵌入(MR-PHE)的新框架,用于零样本病理图像分类。该方法通过多分辨率图像块提取,模仿病理医生的诊断流程,捕捉细粒度细胞特征和整体组织结构。采用混合嵌入策略,结合全局图像嵌入与加权局部嵌入,有效融合局部与全局上下文信息。此外,设计了全面的提示生成与选择框架,通过引入领域特异性同义词和临床相关特征,增强语义理解。基于相似性的块加权机制根据其与类别嵌入的相关性分配注意力权重,突出分类时的关键病灶区域。该方法使用预训练视觉语言模型 CONCH 实现零样本学习,无需领域微调,具备良好可扩展性并减少对大规模标注数据的依赖。实验结果表明,MR-PHE 在多个病理图像数据集上显著提升零样本分类性能,且常优于全监督模型。
原文摘要 · Abstract (English)
Zero-shot learning holds tremendous potential for histopathology image analysis by enabling models to generalize to unseen classes without extensive labeled data. Recent advancements in vision-language models (VLMs) have expanded the capabilities of ZSL, allowing models to perform tasks without task-specific fine-tuning. However, applying VLMs to histopathology presents considerable challenges due to the complexity of histopathological imagery and the nuanced nature of diagnostic tasks. In this paper, we propose a novel framework called Multi-Resolution Prompt-guided Hybrid Embedding (MR-PHE) to address these challenges in zero-shot histopathology image classification. MR-PHE leverages multiresolution patch extraction to mimic the diagnostic workflow of pathologists, capturing both fine-grained cellular details and broader tissue structures critical for accurate diagnosis. We introduce a hybrid embedding strategy that integrates global image embeddings with weighted patch embeddings, effectively combining local and global contextual information. Additionally, we develop a comprehensive prompt generation and selection framework, enriching class descriptions with domain-specific synonyms and clinically relevant features to enhance semantic understanding. A similarity-based patch weighting mechanism assigns attention-like weights to patches based on their relevance to class embeddings, emphasizing diagnostically important regions during classification. Our approach utilizes pretrained VLM, CONCH for ZSL without requiring domain-specific fine-tuning, offering scalability and reducing dependence on large annotated datasets. Experimental results demonstrate that MR-PHE not only significantly improves zero-shot classification performance on histopathology datasets but also often surpasses fully supervised models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。