arXiv:2601.23049cs.AI2026-01ACL被引 3

首个模拟真实医疗计算场景的LLM评测基准,检验模型全流程决策能力。

MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration

  • 通过MCP协议构建多阶段医疗计算任务,模拟真实临床流程。
  • 23个主流模型表现参差,顶级模型仍难应对模糊查询与迭代数据库操作。
  • 适合研究医疗AI推理、工具调用与临床工作流的开发者参考。

医疗计算器是循证临床实践中的核心工具。然而,其实际应用是一个动态的多阶段过程,涉及主动获取电子病历数据、根据场景选择合适计算器以及多步计算,而现有基准仅关注带明确指令的静态单步计算。为此,我们提出MedMCP-Calc,首个基于模型上下文协议(MCP)集成的现实医疗计算场景评测基准。该基准包含跨4个临床领域的118个场景任务,具有模糊任务描述、结构化EHR数据库交互、外部参考检索及过程级评估特点。对23个领先模型的评估显示,即使顶级模型如Claude Opus 4.5也存在明显短板:在模糊查询下难以正确选择端到端工作流所需的计算器,迭代式SQL数据库交互表现差,且显著不愿使用外部工具进行数值计算。不同临床领域间性能差异显著。基于此,我们开发了CalcMate,一个融合场景规划与工具增强的微调模型,在开源模型中达到最先进水平。基准与代码已公开于https://github.com/SPIRAL-MED/MedMCP-Calc。

原文摘要 · Abstract (English)

Medical calculators are fundamental to quantitative, evidence-based clinical practice. However, their real-world use is an adaptive, multi-stage process, requiring proactive EHR data acquisition, scenario-dependent calculator selection, and multi-step computation, whereas current benchmarks focus only on static single-step calculations with explicit instructions. To address these limitations, we introduce MedMCP-Calc, the first benchmark for evaluating LLMs in realistic medical calculator scenarios through Model Context Protocol (MCP) integration. MedMCP-Calc comprises 118 scenario tasks across 4 clinical domains, featuring fuzzy task descriptions mimicking natural queries, structured EHR database interaction, external reference retrieval, and process-level evaluation. Our evaluation of 23 leading models reveals critical limitations: even top performers like Claude Opus 4.5 exhibit substantial gaps, including difficulty selecting appropriate calculators for end-to-end workflows given fuzzy queries, poor performance in iterative SQL-based database interactions, and marked reluctance to leverage external tools for numerical computation. Performance also varies considerably across clinical domains. Building on these findings, we develop CalcMate, a fine-tuned model incorporating scenario planning and tool augmentation, achieving state-of-the-art performance among open-source models. Benchmark and Codes are available in https://github.com/SPIRAL-MED/MedMCP-Calc.

医疗AI大模型评测工具调用临床决策

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。