用少量样本提升病理切片分类准确率，降低提示词波动影响。
MI-VisionShot: Few-shot adaptation of vision-language models for slide-level classification of histopathological images
- 基于视觉语言模型，通过检索关键图像块构建原型分类器。
- 在少样本场景下优于零样本方法，且结果更稳定。
- 适合医学图像分析中数据稀缺的场景，无需重新训练。
视觉-语言监督在利用文本指导学习视觉表征方面取得了显著进展。在数字病理学中，预训练于组织学图像-描述配对数据集的视觉语言模型（VLM）已被用于下游任务，如兴趣区域分类。此前的MI-Zero方法实现了切片级别的零样本迁移，但其性能受文本提示影响较大，存在高变异性。受原型学习启发，我们提出MI-VisionShot，一种无需训练的适应方法，可在少样本学习场景下实现切片级别的标签预测。该框架利用VLM出色的表征能力，在多实例设定下通过检索每张切片中最具判别力的图像块，构建原型分类器。不同实验设置表明，MI-VisionShot在低样本情况下仍能超越零样本迁移，且结果变异性更低。代码将于 soon at https://github.com/cvblab/MIVisionShot。
原文摘要 · Abstract (English)
Vision-language supervision has made remarkable strides in learning visual representations from textual guidance. In digital pathology, vision-language models (VLM), pre-trained on curated datasets of histological image-captions, have been adapted to downstream tasks, such as region of interest classification. Zero-shot transfer for slide-level prediction has been formulated by MI-Zero, but it exhibits high variability depending on the textual prompts. Inspired by prototypical learning, we propose MI-VisionShot, a training-free adaptation method on top of VLMs to predict slide-level labels in few-shot learning scenarios. Our framework takes advantage of the excellent representation learning of VLM to create prototype-based classifiers under a multiple-instance setting by retrieving the most discriminative patches within each slide. Experimentation through different settings shows the ability of MI-VisionShot to surpass zero-shot transfer with lower variability, even in low-shot scenarios. Code coming soon at thttps://github.com/cvblab/MIVisionShot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。