首个面向腹部肿瘤3D影像的医学视觉问答基准，检验大模型临床诊断能力。
Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering
- 构建9262个CT体积、395万图像切片的3D肿瘤问答数据集。
- 模型在测量任务表现尚可，但小病灶识别与临床推理仍严重不足。
- 揭示预训练与视觉模块设计对3D医学图像理解的关键影响。
视觉语言模型（VLMs）在二维视觉任务中展现出潜力，但在三维临床诊断中的适用性仍不明确，因诊断需高精度识别、推理能力及领域知识。为此，我们提出DeepTumorVQA，一个针对腹部肿瘤的3D医学视觉问答基准，涵盖来自17个公开数据集的9,262个CT体积（共370万张切片），包含39.5万道专家级问题，覆盖识别、测量、视觉推理和医学推理四类。该基准引入小肿瘤检测与跨3D解剖结构的临床推理等独特挑战。对四种先进VLM（RadFM、M3D、Merlin、CT-CHAT）的评估显示，当前模型在测量任务表现尚可，但在病灶识别与推理方面仍不达标，无法满足临床需求。关键发现：(1) 大规模多模态预训练显著提升测试性能，使RadFM表现领先；(2) 视觉模块设计与图像预处理方式对3D感知影响显著。为推动医学多模态研究，我们已发布DeepTumorVQA作为严格评估基准：https://github.com/Schuture/DeepTumorVQA。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have shown promise in various 2D visual tasks, yet their readiness for 3D clinical diagnosis remains unclear due to stringent demands for recognition precision, reasoning ability, and domain knowledge. To systematically evaluate these dimensions, we present DeepTumorVQA, a diagnostic visual question answering (VQA) benchmark targeting abdominal tumors in CT scans. It comprises 9,262 CT volumes (3.7M slices) from 17 public datasets, with 395K expert-level questions spanning four categories: Recognition, Measurement, Visual Reasoning, and Medical Reasoning. DeepTumorVQA introduces unique challenges, including small tumor detection and clinical reasoning across 3D anatomy. Benchmarking four advanced VLMs (RadFM, M3D, Merlin, CT-CHAT), we find current models perform adequately on measurement tasks but struggle with lesion recognition and reasoning, and are still not meeting clinical needs. Two key insights emerge: (1) large-scale multimodal pretraining plays a crucial role in DeepTumorVQA testing performance, making RadFM stand out among all VLMs. (2) Our dataset exposes critical differences in VLM components, where proper image preprocessing and design of vision modules significantly affect 3D perception. To facilitate medical multimodal research, we have released DeepTumorVQA as a rigorous benchmark: https://github.com/Schuture/DeepTumorVQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。