测试大模型能否按癌症指南做诊断决策，发现多数模型表现不佳。
MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models
- 从55个NCCN指南构建诊断题库，用LLM生成临床场景。
- 筛选7747道高质量题目，评估25个模型在结构化决策上表现。
- 强调需严格验证模型是否符合临床规程，适合医疗AI安全研究者。
临床指南通常以决策树形式呈现，是循证医学的核心，对确保安全准确的诊断决策至关重要。然而，大语言模型（LLMs）是否能可靠遵循此类结构化流程仍不明确。本文提出MedGUIDE，一个用于评估LLMs在遵循指南基础上做出临床决策能力的新基准。MedGUIDE基于17种癌症类型的55个精心筛选的NCCN决策树，利用LLM生成临床场景，创建大量多选题。通过结合专家标注奖励模型与10项临床及语言标准的LLM裁判集成，进行两阶段质量筛选，最终获得7,747个高质量样本。我们评估了25个涵盖通用、开源及医学专用模型的LLM，发现即使领域专用模型在需要结构化指南遵循的任务中仍常表现不佳。还测试了通过上下文引入指南或持续预训练能否提升性能。结果强调了MedGUIDE在评估模型是否能在真实临床环境中安全运行于程序框架中的重要性。
原文摘要 · Abstract (English)
Clinical guidelines, typically structured as decision trees, are central to evidence-based medical practice and critical for ensuring safe and accurate diagnostic decision-making. However, it remains unclear whether Large Language Models (LLMs) can reliably follow such structured protocols. In this work, we introduce MedGUIDE, a new benchmark for evaluating LLMs on their ability to make guideline-consistent clinical decisions. MedGUIDE is constructed from 55 curated NCCN decision trees across 17 cancer types and uses clinical scenarios generated by LLMs to create a large pool of multiple-choice diagnostic questions. We apply a two-stage quality selection process, combining expert-labeled reward models and LLM-as-a-judge ensembles across ten clinical and linguistic criteria, to select 7,747 high-quality samples. We evaluate 25 LLMs spanning general-purpose, open-source, and medically specialized models, and find that even domain-specific LLMs often underperform on tasks requiring structured guideline adherence. We also test whether performance can be improved via in-context guideline inclusion or continued pretraining. Our findings underscore the importance of MedGUIDE in assessing whether LLMs can operate safely within the procedural frameworks expected in real-world clinical settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。