构建医学专家级推理评测基准，含图文多模态题库。
MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
- 设计17个专科、11个系统共4460道高难度医学题
- 引入真实临床图像与病历数据，提升评估真实性
- 专为测试大模型医学推理能力而设，适合临床AI研究者
我们提出MedXpertQA，一个高度挑战且全面的基准，用于评估医学专家级知识与高级推理能力。该基准包含4,460道题目，覆盖17个专科和11个身体系统，分为文本-文本（Text）与多模态（MM）两个子集。其中，MM子集引入了具有多样图像和丰富临床信息（包括患者记录、检查结果）的专家级考试题，区别于传统基于图像描述生成的简单问答对。MedXpertQA通过严格筛选与增强，解决现有基准（如MedQA）难度不足的问题，并融入专科认证考题以提高临床相关性与完整性。我们采用数据合成方法降低数据泄露风险，并经过多轮专家评审确保准确性与可靠性。我们在该基准上评估了18个领先模型。医学决策紧密关联现实世界，是检验推理能力的理想场景。为此，我们还开发了一个面向推理的子集，以支持o1类模型的评估。代码与数据已开源：https://github.com/TsinghuaC3I/MedXpertQA。
原文摘要 · Abstract (English)
We introduce MedXpertQA, a highly challenging and comprehensive benchmark to evaluate expert-level medical knowledge and advanced reasoning. MedXpertQA includes 4,460 questions spanning 17 specialties and 11 body systems. It includes two subsets, Text for text evaluation and MM for multimodal evaluation. Notably, MM introduces expert-level exam questions with diverse images and rich clinical information, including patient records and examination results, setting it apart from traditional medical multimodal benchmarks with simple QA pairs generated from image captions. MedXpertQA applies rigorous filtering and augmentation to address the insufficient difficulty of existing benchmarks like MedQA, and incorporates specialty board questions to improve clinical relevance and comprehensiveness. We perform data synthesis to mitigate data leakage risk and conduct multiple rounds of expert reviews to ensure accuracy and reliability. We evaluate 18 leading models on \benchmark. Moreover, medicine is deeply connected to real-world decision-making, providing a rich and representative setting for assessing reasoning abilities beyond mathematics and code. To this end, we develop a reasoning-oriented subset to facilitate the assessment of o1-like models. Code and data are available at: https://github.com/TsinghuaC3I/MedXpertQA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。