构建首个大规模医疗计算评估基准,提升大模型临床数值推理能力。
MedCalc-Eval and MedCalc-Env: Advancing Medical Calculation Capabilities of Large Language Models
- 设计双类计算任务:公式型与评分系统,覆盖多专科临床场景。
- 在700+任务上验证模型,实现数值敏感度与公式选择的显著提升。
- 适合医疗AI研究者、临床决策系统开发者使用。
随着大语言模型进入医疗领域,现有评估多聚焦于问答与描述性推理,忽视了临床决策中至关重要的定量推理。现有数据集如MedCalc-Bench涵盖任务有限,难以反映真实计算场景。本文提出MedCalc-Eval,目前最大的医疗计算能力评估基准,包含700余项任务,分为公式类(如Cockcroft-Gault、BMI、BSA)与规则评分类(如Apgar、Glasgow Coma Scale),覆盖内科、外科、儿科、心血管等多个专科,提供更全面且更具挑战性的评测环境。为进一步提升性能,我们构建MedCalc-Env,基于InternBootcamp框架的强化学习环境,支持多步临床推理与规划。在该环境中微调Qwen2.5-32B模型,在MedCalc-Eval上取得当前最优表现,尤其在数值敏感度、公式选择与推理鲁棒性方面进步显著。现存挑战包括单位转换、多条件逻辑与上下文理解。代码与数据集已开源:https://github.com/maokangkun/MedCalc-Eval。
原文摘要 · Abstract (English)
As large language models (LLMs) enter the medical domain, most benchmarks evaluate them on question answering or descriptive reasoning, overlooking quantitative reasoning critical to clinical decision-making. Existing datasets like MedCalc-Bench cover few calculation tasks and fail to reflect real-world computational scenarios. We introduce MedCalc-Eval, the largest benchmark for assessing LLMs' medical calculation abilities, comprising 700+ tasks across two types: equation-based (e.g., Cockcroft-Gault, BMI, BSA) and rule-based scoring systems (e.g., Apgar, Glasgow Coma Scale). These tasks span diverse specialties including internal medicine, surgery, pediatrics, and cardiology, offering a broader and more challenging evaluation setting. To improve performance, we further develop MedCalc-Env, a reinforcement learning environment built on the InternBootcamp framework, enabling multi-step clinical reasoning and planning. Fine-tuning a Qwen2.5-32B model within this environment achieves state-of-the-art results on MedCalc-Eval, with notable gains in numerical sensitivity, formula selection, and reasoning robustness. Remaining challenges include unit conversion, multi-condition logic, and contextual understanding. Code and datasets are available at https://github.com/maokangkun/MedCalc-Eval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。