构建多维度医学影像退化评估基准,检验大模型在真实临床环境中的可靠性。
MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations
- 设计覆盖18类退化、30维能力的多维度评估体系
- 发现模型在退化下准确率下降,但自信度仍高,存在认知偏差
- 适合医疗AI可信性研究者和临床部署开发者参考
尽管在标准基准上表现优异,多模态大语言模型(MLLMs)在真实临床环境中面临医学图像质量退化的严峻挑战。现有评估基准存在两大局限:(1)缺乏跨医学图像质量梯度的大规模、多维度评估;(2)缺乏系统性的置信度校准分析。为此,我们提出MedQ-Deg,一个全面评估医学MLLM在图像质量退化下的基准。MedQ-Deg涵盖18种不同退化类型、30个细粒度能力维度、7种成像模态,共包含24,894个问答对。每种退化设置3个严重程度等级,并由放射科专家校准。我们进一步引入校准偏移(Calibration Shift)指标,量化模型感知置信度与实际性能之间的差距,以评估其在退化下的元认知可靠性。对40个主流MLLM的全面评估揭示:(1)模型性能随退化严重程度系统性下降;(2)模型普遍存在人工智能版邓宁-克鲁格效应,即使准确率大幅下降仍保持过高自信;(3)模型在不同能力维度、成像模态和退化类型间表现出显著差异的行为模式。我们希望MedQ-Deg推动医学MLLM向真实临床实践中更稳健、可信的方向发展。
原文摘要 · Abstract (English)
Despite impressive performance on standard benchmarks, multimodal large language models (MLLMs) face critical challenges in real-world clinical environments where medical images inevitably suffer various quality degradations. Existing benchmarks exhibit two key limitations: (1) absence of large-scale, multidimensional assessment across medical image quality gradients and (2) no systematic confidence calibration analysis. To address these gaps, we present MedQ-Deg, a comprehensive benchmark for evaluating medical MLLMs under image quality degradations. MedQ-Deg provides multi-dimensional evaluation spanning 18 distinct degradation types, 30 fine-grained capability dimensions, and 7 imaging modalities, with 24,894 question-answer pairs. Each degradation is implemented at 3 severity degrees, calibrated by expert radiologists. We further introduce Calibration Shift metric, which quantifies the gap between a model's perceived confidence and actual performance to assess metacognitive reliability under degradation. Our comprehensive evaluation of 40 mainstream MLLMs reveals several critical findings: (1) overall model performance degrades systematically as degradation severity increases, (2) models universally exhibit the AI Dunning-Kruger Effect, maintaining inappropriately high confidence despite severe accuracy collapse, and (3) models display markedly differentiated behavioral patterns across capability dimensions, imaging modalities, and degradation types. We hope MedQ-Deg drives progress toward medical MLLMs that are robust and trustworthy in real clinical practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。