评估视觉语言模型在病理图像细胞分类中的表现，发现其需少量样本提示才有效。
Evaluating Cell Type Inference in Vision Language Models Under Varying Visual Context
- 用零样本和少样本提示测试主流VLMs在病理图像上的表现
- 少样本提示使性能显著提升，但仍不及专门训练的CNN
- 适合研究AI在医学图像分析中应用潜力的学者参考
视觉语言模型（VLMs）随着大语言模型的发展迅速进步。本研究评估了通过API访问的代表性生成式VLMs（如GPT-4.1和Gemini 2.5 Pro）在组织病理学图像分类任务中的表现，包括细胞类型识别。利用来自公开和私有来源的多样化数据集，采用零样本与少样本提示方法评估VLM性能，并与自训练卷积神经网络（CNNs）进行对比。结果表明，尽管少样本提示相比零样本显著提升了VLM表现（基于卡帕系数，p ≈ 1.005 × 10⁻⁵），当前通用型VLM在多数任务上仍逊于监督训练的CNN。该工作凸显了将现有VLM应用于病理等专业领域时，通过上下文学习所展现的潜力与局限性。所有代码及复现说明可从https://www.github.com/a12dongithub/VLMCCE获取。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have rapidly advanced alongside Large Language Models (LLMs). This study evaluates the capabilities of prominent generative VLMs, such as GPT-4.1 and Gemini 2.5 Pro, accessed via APIs, for histopathology image classification tasks, including cell typing. Using diverse datasets from public and private sources, we apply zero-shot and one-shot prompting methods to assess VLM performance, comparing them against custom-trained Convolutional Neural Networks (CNNs). Our findings demonstrate that while one-shot prompting significantly improves VLM performance over zero-shot ($p \approx 1.005 \times 10^{-5}$ based on Kappa scores), these general-purpose VLMs currently underperform supervised CNNs on most tasks. This work underscores both the promise and limitations of applying current VLMs to specialized domains like pathology via in-context learning. All code and instructions for reproducing the study can be accessed from the repository https://www.github.com/a12dongithub/VLMCCE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。