让大模型分层理解病理切片,像医生一样逐级分析细胞、组织和整体。
MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding
- 将切片按细胞、区块、区域、整体四尺度建模,对齐语言与视觉证据
- 在13个基准上实现新最佳性能,诊断相关区域定位更精准
- 适合医学影像分析、智能病理诊断等研究者使用
全切片图像(WSI)具有层级结构,诊断信息来自细胞形态、区域组织布局及全局上下文。现有计算病理学多模态大模型通常将整张切片压缩为单一嵌入,阻碍细粒度定位,且忽略病理科医生跨尺度整合证据的过程。我们提出MLLM-HWSI,一种分层的全切片级多模态大模型,将视觉特征与病理语言在四个尺度对齐:细胞如词、区块如短语、区域如句子、全切片如段落,支持可解释的证据驱动推理。该模型通过尺度特定投影器生成多尺度嵌入,并联合施加层次对比损失与跨尺度一致性损失,保持从细胞到切片的语义连贯性。利用轻量级'细胞-细胞注意力融合(CCAF)'Transformer,将分割后的细胞嵌入聚合为每区块紧凑的细胞标记。多尺度标记与文本标记融合后输入指令微调的大语言模型,完成开放式推理、问答、报告生成与图像描述任务。三阶段训练后,MLLM-HWSI在六个计算病理任务的13个切片级基准上达到新最优结果。通过语言与多尺度视觉证据对齐,模型输出准确且可解释,模拟真实诊断流程,推动全面的切片理解。代码已开源。
原文摘要 · Abstract (English)
Whole Slide Images (WSIs) exhibit hierarchical structure, where diagnostic information emerges from cellular morphology, regional tissue organization, and global context. Existing Computational Pathology (CPath) Multimodal Large Language Models (MLLMs) typically compress an entire WSI into a single embedding, which hinders fine-grained grounding and ignores how pathologists synthesize evidence across different scales. We introduce \textbf{MLLM-HWSI}, a Hierarchical WSI-level MLLM that aligns visual features with pathology language at four distinct scales, cell as word, patch as phrase, region as sentence, and WSI as paragraph to support interpretable evidence-grounded reasoning. MLLM-HWSI decomposes each WSI into multi-scale embeddings with scale-specific projectors and jointly enforces (i) a hierarchical contrastive objective and (ii) a cross-scale consistency loss, preserving semantic coherence from cells to the WSI. We compute diagnostically relevant patches and aggregate segmented cell embeddings into a compact cellular token per-patch using a lightweight \textit{Cell-Cell Attention Fusion (CCAF)} transformer. The projected multi-scale tokens are fused with text tokens and fed to an instruction-tuned LLM for open-ended reasoning, VQA, report, and caption generation tasks. Trained in three stages, MLLM-HWSI achieves new SOTA results on 13 WSI-level benchmarks across six CPath tasks. By aligning language with multi-scale visual evidence, MLLM-HWSI provides accurate, interpretable outputs that mirror diagnostic workflows and advance holistic WSI understanding. Code is available at: \href{https://github.com/BasitAlawode/HWSI-MLLM}{GitHub}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。