测试大模型在心电图诊断中的推理能力，发现其多步逻辑链几乎失效。
ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation
- 构建6400+样本的多轮评估框架，检验心电图诊断的逐步推理。
- 顶尖模型仅6%完成完整推理链，无法关联视觉证据与诊断结论。
- 揭示当前医疗AI依赖表面特征，适合关注医学AI可解释性的研究者。
尽管多模态大语言模型在心电图自动解读中表现优异，但尚不清楚它们是否真正执行逐步推理，还是仅依赖表层视觉线索。为此，我们提出ECG-Reasoning-Benchmark，一个包含超过6,400个样本的多轮评估框架，系统性地检验17种核心心电图诊断中的逐步推理能力。对前沿模型的全面评估显示，其在多步逻辑推导上存在严重缺陷：尽管具备获取临床诊断标准的医学知识，但在保持完整推理链方面成功率极低（6%完成率），主要失败在于无法将具体心电图发现与实际信号中的视觉证据有效关联。结果表明，当前多模态大语言模型绕过了真实的视觉解析过程，暴露出现有训练范式的重大缺陷，凸显了发展以推理为中心的医疗人工智能的迫切需求。代码与数据见https://github.com/Jwoo5/ecg-reasoning-benchmark。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation, it remains unclear whether they genuinely perform actual step-by-step reasoning or just rely on superficial visual cues. To investigate this, we introduce \textbf{ECG-Reasoning-Benchmark}, a novel multi-turn evaluation framework comprising over 6,400 samples to systematically assess step-by-step reasoning across 17 core ECG diagnoses. Our comprehensive evaluation of state-of-the-art models reveals a critical failure in executing multi-step logical deduction. Although models possess the medical knowledge to retrieve clinical criteria for a diagnosis, they exhibit near-zero success rates (6% Completion) in maintaining a complete reasoning chain, primarily failing to ground the corresponding ECG findings to the actual visual evidence in the ECG signal. These results demonstrate that current MLLMs bypass actual visual interpretation, exposing a critical flaw in existing training paradigms and underscoring the necessity for robust, reasoning-centric medical AI. The code and data are available at https://github.com/Jwoo5/ecg-reasoning-benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。