arXiv:2603.07769cs.CV2026-03

构建多维度医学影像退化评估基准,检验大模型在真实临床环境中的可靠性。

MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations

  • 设计覆盖18类退化、30维能力的多维度评估体系
  • 发现模型在退化下准确率下降,但自信度仍高,存在认知偏差
  • 适合医疗AI可信性研究者和临床部署开发者参考

尽管在标准基准上表现优异,多模态大语言模型(MLLMs)在真实临床环境中面临医学图像质量退化的严峻挑战。现有评估基准存在两大局限:(1)缺乏跨医学图像质量梯度的大规模、多维度评估;(2)缺乏系统性的置信度校准分析。为此,我们提出MedQ-Deg,一个全面评估医学MLLM在图像质量退化下的基准。MedQ-Deg涵盖18种不同退化类型、30个细粒度能力维度、7种成像模态,共包含24,894个问答对。每种退化设置3个严重程度等级,并由放射科专家校准。我们进一步引入校准偏移(Calibration Shift)指标,量化模型感知置信度与实际性能之间的差距,以评估其在退化下的元认知可靠性。对40个主流MLLM的全面评估揭示:(1)模型性能随退化严重程度系统性下降;(2)模型普遍存在人工智能版邓宁-克鲁格效应,即使准确率大幅下降仍保持过高自信;(3)模型在不同能力维度、成像模态和退化类型间表现出显著差异的行为模式。我们希望MedQ-Deg推动医学MLLM向真实临床实践中更稳健、可信的方向发展。

原文摘要 · Abstract (English)

Despite impressive performance on standard benchmarks, multimodal large language models (MLLMs) face critical challenges in real-world clinical environments where medical images inevitably suffer various quality degradations. Existing benchmarks exhibit two key limitations: (1) absence of large-scale, multidimensional assessment across medical image quality gradients and (2) no systematic confidence calibration analysis. To address these gaps, we present MedQ-Deg, a comprehensive benchmark for evaluating medical MLLMs under image quality degradations. MedQ-Deg provides multi-dimensional evaluation spanning 18 distinct degradation types, 30 fine-grained capability dimensions, and 7 imaging modalities, with 24,894 question-answer pairs. Each degradation is implemented at 3 severity degrees, calibrated by expert radiologists. We further introduce Calibration Shift metric, which quantifies the gap between a model's perceived confidence and actual performance to assess metacognitive reliability under degradation. Our comprehensive evaluation of 40 mainstream MLLMs reveals several critical findings: (1) overall model performance degrades systematically as degradation severity increases, (2) models universally exhibit the AI Dunning-Kruger Effect, maintaining inappropriately high confidence despite severe accuracy collapse, and (3) models display markedly differentiated behavioral patterns across capability dimensions, imaging modalities, and degradation types. We hope MedQ-Deg drives progress toward medical MLLMs that are robust and trustworthy in real clinical practice.

医学AI模型评估可信性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。