arXiv:2504.00784cs.CVcs.LG2025-04

用轻量模块增强视觉Transformer,提升病理切片细胞分割精度

CellVTA: Enhancing Vision Foundation Models for Accurate Cell Segmentation and Classification

  • 在ViT中引入CNN适配器,注入高分辨率空间信息
  • CoNIC和PanNuke数据集上分别达0.538和0.506 mPQ
  • 无需重训练,可直接接入预训练模型,适合医疗图像研究者

细胞实例分割是数字病理学中的基础任务,具有广泛临床应用。近期基于视觉变换器(ViTs)的视觉基础模型在病理图像分析中取得显著进展,但在细胞实例分割上的提升仍有限。主要挑战源于ViT的分块处理会大幅降低输入图像的空间分辨率,导致小细胞和密集排列细胞的分割质量不佳。为此,我们提出CellVTA(带适配器的细胞视觉变换器),通过引入基于CNN的适配器模块,从输入图像中提取高分辨率空间特征,并通过交叉注意力机制注入ViT。该方法保持了ViT的核心结构,可无缝集成至预训练基础模型。大量实验表明,CellVTA在CoNIC数据集上达到0.538 mPQ,PanNuke数据集上达到0.506 mPQ,显著优于当前最优方法。消融实验验证了该方法在参数微调策略中的优越性,包括仅微调解码器和全模型微调。代码与模型已开源。

原文摘要 · Abstract (English)

Cell instance segmentation is a fundamental task in digital pathology with broad clinical applications. Recently, vision foundation models, which are predominantly based on Vision Transformers (ViTs), have achieved remarkable success in pathology image analysis. However, their improvements in cell instance segmentation remain limited. A key challenge arises from the tokenization process in ViTs, which substantially reduces the spatial resolution of input images, leading to suboptimal segmentation quality, especially for small and densely packed cells. To address this problem, we propose CellVTA (Cell Vision Transformer with Adapter), a novel method that improves the performance of vision foundation models for cell instance segmentation by incorporating a CNN-based adapter module. This adapter extracts high-resolution spatial information from input images and injects it into the ViT through a cross-attention mechanism. Our method preserves the core architecture of ViT, ensuring seamless integration with pretrained foundation models. Extensive experiments show that CellVTA achieves 0.538 mPQ on the CoNIC dataset and 0.506 mPQ on the PanNuke dataset, which significantly outperforms the state-of-the-art cell segmentation methods. Ablation studies confirm the superiority of our approach over other fine-tuning strategies, including decoder-only fine-tuning and full fine-tuning. Our code and models are publicly available at https://github.com/JieZheng-ShanghaiTech/CellVTA.

细胞分割视觉Transformer病理图像适配器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。