arXiv:2604.16506cs.CVcs.CL2026-04被引 6

评测大模型跨多图医学诊断能力,发现证据整合是关键瓶颈。

Medical thinking with multiple images

论文配图:Medical thinking with multiple images
图 1 · 摘自论文原文
  • 构建多图像医疗问答数据集,要求模型跨图整合证据并逐步推理。
  • 顶尖模型准确率仅57%,超70%错误源于图像理解与跨图对齐失败。
  • 提升性能需先强化视觉定位,长推理链对弱基础无益反而放大错误。

大型语言模型在多项医学问答基准上表现良好,但真实临床推理常需整合多幅图像而非单图解读。我们提出MedThinkVQA,一个专家标注的多图思维基准,要求模型解析每张图像、融合跨视图证据,并在中间监督和逐步评估下回答诊断问题。数据集包含8,067个病例(含720个测试用例),平均每例6.62幅图像,远超以往研究(最高1.43幅/例)。测试集上,最佳闭源模型Claude-4.6-Opus、Gemini-3-Pro、GPT-5.2-xhigh准确率分别为57.2%、55.3%、54.9%;GPT-5-mini和GPT-5-nano分别为39.7%和30.8%。开源模型中领先者Qwen3.5-397B-A17B达52.2%,Qwen3.5-27B为50.6%。分析表明,可靠跨图证据整合是主要瓶颈:模型常在高阶推理前未能有效提取、对齐与组合多视图信息。提供专家标注的单图提示与跨图摘要可提升性能,而用自生成中间步骤则降低准确率。逐步分析显示,超过70%错误源于图像阅读与跨图整合。扩展实验表明,当视觉定位不可靠时,增加推理计算仅带来有限或不稳定的收益,甚至放大误读信号。结果表明,核心挑战并非推理长度,而是建立可靠机制以实现真实世界多模态临床输入中的证据定位、对齐与融合。

原文摘要 · Abstract (English)

Large language models perform well on many medical QA benchmarks, but real clinical reasoning often requires integrating evidence across multiple images rather than interpreting a single view. We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images, where models must interpret each image, combine cross-view evidence, and answer diagnostic questions with intermediate supervision and step-level evaluation. The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case, substantially denser than prior work, whose expert-level benchmarks use at most 1.43 images per case. On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy, while GPT-5-mini and GPT-5-nano reach 39.7% and 30.8%. Strong open-source models lag behind, led by Qwen3.5-397B-A17B at 52.2% and Qwen3.5-27B at 50.6%. Further analysis identifies grounded multi-image reasoning as the main bottleneck: models often fail to extract, align, and compose evidence across views before higher-level inference can help. Providing expert single-image cues and cross-image summaries improves performance, whereas replacing them with self-generated intermediates reduces accuracy. Step-level analysis shows that over 70% of errors arise from image reading and cross-view integration. Scaling results further show that additional inference-time computation helps only when visual grounding is already reliable; when early evidence extraction is weak, longer reasoning yields limited or unstable gains and can amplify misread cues. These results suggest that the key challenge is not reasoning length alone, but reliable mechanisms for grounding, aligning, and composing distributed evidence across real-world multimodal clinical inputs.

多图推理医学AI视觉定位模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。