用属性自动生成提示词,让视觉语言模型零样本检测病理切片中的细胞核。
AttriPrompter: Auto-Prompting with Attribute Semantics for Zero-shot Nuclei Detection via Visual-Language Pre-trained Models
- 基于属性生成与排序的自动提示框架,避免人工设计提示词。
- 在高密度细胞核场景下实现93.6%的检测准确率,优于现有无监督方法。
- 适用于医学图像零样本检测,尤其适合缺乏标注数据的研究者。
大规模视觉-语言预训练模型(VLPMs)在自然场景物体检测中通过文本提示展现卓越性能。然而,其在组织病理图像零样本细胞核检测中的应用仍不充分,主要因医学图像特性与网络来源图文对存在显著差异。本文研究了面向零样本细胞核检测的物体级VLPM——GLIP的潜力。提出创新的自动提示管道AttriPrompter,包含属性生成、属性增强和相关性排序,避免主观手动设计提示。AttriPrompter利用VLPM的文本-图像对齐能力生成语义丰富的文本提示,输入GLIP进行初始零样本检测。此外,提出自训练知识蒸馏框架,以GLIP初始预测作为伪标签,缓解高细胞密度带来的漏检、误检和重叠问题。该方法在无标注数据下表现优异,超越所有现有无监督方法,展现出极强泛化性。本工作揭示了在自然图像-文本对上预训练的VLPM在医疗领域下游任务中的巨大潜力。代码将发布于https://github.com/wuyongjianCODE/AttriPrompter。
原文摘要 · Abstract (English)
Large-scale visual-language pre-trained models (VLPMs) have demonstrated exceptional performance in downstream object detection through text prompts for natural scenes. However, their application to zero-shot nuclei detection on histopathology images remains relatively unexplored, mainly due to the significant gap between the characteristics of medical images and the web-originated text-image pairs used for pre-training. This paper aims to investigate the potential of the object-level VLPM, Grounded Language-Image Pre-training (GLIP), for zero-shot nuclei detection. Specifically, we propose an innovative auto-prompting pipeline, named AttriPrompter, comprising attribute generation, attribute augmentation, and relevance sorting, to avoid subjective manual prompt design. AttriPrompter utilizes VLPMs' text-to-image alignment to create semantically rich text prompts, which are then fed into GLIP for initial zero-shot nuclei detection. Additionally, we propose a self-trained knowledge distillation framework, where GLIP serves as the teacher with its initial predictions used as pseudo labels, to address the challenges posed by high nuclei density, including missed detections, false positives, and overlapping instances. Our method exhibits remarkable performance in label-free nuclei detection, outperforming all existing unsupervised methods and demonstrating excellent generality. Notably, this work highlights the astonishing potential of VLPMs pre-trained on natural image-text pairs for downstream tasks in the medical field as well. Code will be released at https://github.com/wuyongjianCODE/AttriPrompter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。