用可学习的钩子令牌高效捕捉病理图像上下文,提升模型性能与可解释性。
HookMIL: Revisiting Context Modeling in Multiple Instance Learning for Computational Pathology
- 引入可学习钩子令牌,通过多模态初始化融合视觉、文本和空间先验。
- 线性复杂度注意力机制使训练更快,4个数据集上均达最优表现。
- 适合需要高效、可解释的弱监督病理分析的研究者使用。
多实例学习(MIL)已实现全切片图像(WSI)的弱监督病理分析。但传统MIL方法常丢失关键上下文信息,而基于Transformer的变体虽更具表达力,却存在二次复杂度和冗余计算问题。为此,本文提出HookMIL,一种具上下文感知能力且计算高效的MIL框架,利用紧凑可学习的钩子令牌进行结构化上下文聚合。这些令牌可从(i)关键切片视觉特征、(ii)视觉-语言病理模型的文本嵌入、(iii)空间转录组-视觉模型的空间特征初始化。多模态初始化使钩子令牌融合丰富文本与空间先验,加速收敛并提升表示质量。训练中,钩子令牌通过双向注意力与实例交互,实现线性复杂度;为促进专业化,引入钩子多样性损失,促使每个令牌关注不同组织学模式;此外,钩子间通信机制优化上下文交互,减少冗余。在四个公开病理数据集上的实验表明,HookMIL达到当前最优性能,兼具更高效率与可解释性。代码已开源。
原文摘要 · Abstract (English)
Multiple Instance Learning (MIL) has enabled weakly supervised analysis of whole-slide images (WSIs) in computational pathology. However, traditional MIL approaches often lose crucial contextual information, while transformer-based variants, though more expressive, suffer from quadratic complexity and redundant computations. To address these limitations, we propose HookMIL, a context-aware and computationally efficient MIL framework that leverages compact, learnable hook tokens for structured contextual aggregation. These tokens can be initialized from (i) key-patch visual features, (ii) text embeddings from vision-language pathology models, and (iii) spatially grounded features from spatial transcriptomics-vision models. This multimodal initialization enables Hook Tokens to incorporate rich textual and spatial priors, accelerating convergence and enhancing representation quality. During training, Hook tokens interact with instances through bidirectional attention with linear complexity. To further promote specialization, we introduce a Hook Diversity Loss that encourages each token to focus on distinct histopathological patterns. Additionally, a hook-to-hook communication mechanism refines contextual interactions while minimizing redundancy. Extensive experiments on four public pathology datasets demonstrate that HookMIL achieves state-of-the-art performance, with improved computational efficiency and interpretability. Codes are available at https://github.com/lingxitong/HookMIL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。