首个覆盖多阶段临床任务的脑影像多模态评测基准,填补AI模型评估空白。
OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks
- 构建15类影像模态、9527个问答对的多模态评测集
- 专业放射科医生验证15个临床流程,真实模拟诊疗场景
- 揭示大模型在术前推理等复杂任务中与医生差距显著
脑影像分析对诊断和治疗脑疾病至关重要,多模态大语言模型(MLLMs)正日益提供支持。然而,现有脑影像视觉问答(VQA)基准或覆盖模态有限,或仅提供粗粒度病理描述,难以全面评估MLLMs在完整临床流程中的表现。为此,我们提出OmniBrainBench,首个专为评估脑影像分析中MLLM多模态理解能力而设计的综合性多模态VQA基准,支持闭合与开放式评估。该基准涵盖来自30个权威医学来源的15种不同脑影像模态,共生成9,527个经验证的VQA对和31,706张图像。它模拟真实临床工作流程,包含15项经放射科医生严格验证的多阶段临床任务。对24个前沿模型(包括开源通用型、医疗专用型及商用模型)的评估显示,尽管商用模型如GPT-5(准确率63.37%)表现最优,但仍远低于医生水平(91.35%),医疗类模型在闭合与开放式任务间差异大;开源通用模型整体落后但部分任务突出,所有模型在复杂术前推理任务中均表现不足,暴露出显著的视觉到临床认知鸿沟。OmniBrainBench确立了脑影像分析中评估MLLMs的新标准,凸显其与临床医生之间的差距。我们已公开发布该基准。
原文摘要 · Abstract (English)
Brain imaging analysis is crucial for diagnosing and treating brain disorders, and multimodal large language models (MLLMs) are increasingly supporting it. However, current brain imaging visual question-answering (VQA) benchmarks either cover a limited number of imaging modalities or are restricted to coarse-grained pathological descriptions, hindering a comprehensive assessment of MLLMs across the full clinical continuum. To address these, we introduce OmniBrainBench, the first comprehensive multimodal VQA benchmark specifically designed to assess the multimodal comprehension capabilities of MLLMs in brain imaging analysis with closed- and open-ended evaluations. OmniBrainBench comprises 15 distinct brain imaging modalities collected from 30 verified medical sources, yielding 9,527 validated VQA pairs and 31,706 images. It simulates clinical workflows and encompasses 15 multi-stage clinical tasks rigorously validated by a professional radiologist. Evaluations of 24 state-of-the-art models, including open-source general-purpose, medical, and proprietary MLLMs, highlight the substantial challenges posed by OmniBrainBench. Experiments reveal that proprietary MLLMs like GPT-5 (63.37%) outperform others yet lag far behind physicians (91.35%), while medical ones show wide variance in closed- and open-ended VQA. Open-source general-purpose MLLMs generally trail but excel in specific tasks, and all ones fall short in complex preoperative reasoning, revealing a critical visual-to-clinical gap. OmniBrainBench establishes a new standard to assess MLLMs in brain imaging analysis, highlighting the gaps against physicians. We publicly release our benchmark at link.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。