用指令微调视觉语言模型，实现医学图像中病灶的定位、计数与检测一体化。
Point, Detect, Count: Multi-Task Medical Image Understanding with Instruction-Tuned Vision-Language Models
- 通过指令提示将多任务转化为视觉语言推理，统一处理定位、计数与检测。
- 在多任务训练下，计数任务的平均绝对误差降低，匹配准确率提升。
- 输出可解释且结构化，适合临床医生协同诊断场景使用。
我们研究了将视觉语言模型（VLMs）微调用于多任务医学图像理解，聚焦于医学图像中病灶的检测、定位与计数。目标是评估指令微调的VLM是否能同时提升这些任务，以增强诊断准确性和效率。基于包含内窥镜（息肉与器械）和显微镜（精子细胞）标注的多模态数据集MedMultiPoints，我们将每项任务重构为适合视觉语言推理的指令提示。采用低秩适配（LoRA）对Qwen2.5-VL-7B-Instruct进行多任务组合微调。结果表明，多任务训练提升了模型鲁棒性与准确性：例如，在计数+定位任务中，计数平均绝对误差（MAE）下降，匹配准确率上升。但存在权衡，如零样本情况下的定位预测增多，表明边缘案例可靠性下降。本研究展示了通过提示驱动微调，将通用VLM适配至专业医学任务的潜力。该方法模拟放射科医生同步定位、计数与描述发现的临床流程，证明了VLM可学习复合诊断推理模式。模型生成可解释、结构化输出，为可解释且多功能的医疗AI提供了重要进展。代码、模型权重与脚本将开源发布于https://github.com/simula/PointDetectCount。
原文摘要 · Abstract (English)
We investigate fine-tuning Vision-Language Models (VLMs) for multi-task medical image understanding, focusing on detection, localization, and counting of findings in medical images. Our objective is to evaluate whether instruction-tuned VLMs can simultaneously improve these tasks, with the goal of enhancing diagnostic accuracy and efficiency. Using MedMultiPoints, a multimodal dataset with annotations from endoscopy (polyps and instruments) and microscopy (sperm cells), we reformulate each task into instruction-based prompts suitable for vision-language reasoning. We fine-tune Qwen2.5-VL-7B-Instruct using Low-Rank Adaptation (LoRA) across multiple task combinations. Results show that multi-task training improves robustness and accuracy. For example, it reduces the Count Mean Absolute Error (MAE) and increases Matching Accuracy in the Counting + Pointing task. However, trade-offs emerge, such as more zero-case point predictions, indicating reduced reliability in edge cases despite overall performance gains. Our study highlights the potential of adapting general-purpose VLMs to specialized medical tasks via prompt-driven fine-tuning. This approach mirrors clinical workflows, where radiologists simultaneously localize, count, and describe findings - demonstrating how VLMs can learn composite diagnostic reasoning patterns. The model produces interpretable, structured outputs, offering a promising step toward explainable and versatile medical AI. Code, model weights, and scripts will be released for reproducibility at https://github.com/simula/PointDetectCount.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。