首个面向麻醉学推理的综合数据集与评测基准,助力大模型精准决策。
AnesSuite: A Comprehensive Benchmark and Dataset Suite for Anesthesiology Reasoning in LLMs
- 构建三层次评测框架,涵盖事实检索、混合推理与复杂决策
- 引入3个训练数据集,支持持续预训练、监督微调与可验证奖励强化学习
- 推出首个麻醉推理基线模型,性能媲美大型模型,通用医学能力也提升
大语言模型在医疗领域的应用备受关注,但其在麻醉学等专业领域中的推理能力仍待深入探索。为此,我们提出AnesSuite,首个专为麻醉学推理设计的综合性数据集与评测套件。该套件包含AnesBench,一个覆盖三层次推理能力的评估基准:事实检索(系统1)、混合推理(系统1.x)和复杂决策(系统2)。同时配套三个训练数据集,支持持续预训练(CPT)、监督微调(SFT)和基于可验证奖励的强化学习(RLVR)。基于此,我们构建了首个麻醉学推理基线模型集合Morpheus。尽管仅通过有限的SFT和组相对策略优化(GRPO)训练,Morpheus不仅在麻醉学任务上表现优异,接近更大规模模型,还在通用医学及跨领域基准中展现出更强的推理能力。通过全面实验,我们分析了模型特性、训练策略与数据对麻醉学推理性能的影响。AnesSuite与Morpheus将开源于https://github.com/MiliLab/AnesSuite。
原文摘要 · Abstract (English)
The application of large language models (LLMs) in the medical field has garnered significant attention, yet their reasoning capabilities in more specialized domains like anesthesiology remain underexplored. To bridge this gap, we introduce AnesSuite, the first comprehensive dataset suite specifically designed for anesthesiology reasoning in LLMs. The suite features AnesBench, an evaluation benchmark tailored to assess anesthesiology-related reasoning across three levels: factual retrieval (System 1), hybrid reasoning (System 1.x), and complex decision-making (System 2). Alongside this benchmark, the suite includes three training datasets that provide an infrastructure for continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning with verifiable rewards (RLVR). Leveraging this suite, we develop Morpheus, the first baseline model collection for anesthesiology reasoning. Despite undergoing limited training with SFT and group relative policy optimization (GRPO), Morpheus not only achieves substantial improvements in anesthesiology that rival larger-scale models, but also demonstrates enhanced reasoning capabilities across general medical and broad-domain benchmarks. Furthermore, through comprehensive evaluations and experiments, we analyze the key factors influencing anesthesiology reasoning performance, including model characteristics, training strategies and training data. Both AnesSuite and Morpheus will be open-sourced at https://github.com/MiliLab/AnesSuite.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。