构建首个法语医学问答数据集，评估模型在真实临床场景下的知识与推理能力。
MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation
- 基于41个医学领域的真实法语医考题，构建3.2万条问答数据。
- 包含单选、多选和开放题三类任务，区分理解与推理类型。
- 揭示大模型在医学事实记忆与推理间存在显著能力差距。
本文提出MediQAl，一个用于评估语言模型在真实临床场景中医学知识回忆与推理能力的法语医学问答数据集。该数据集包含来自41个医学领域的32,603道问题，涵盖三类任务：(i) 单一答案的选择题，(ii) 多个答案的选择题，(iii) 简短回答的开放题。每道题均标注为理解或推理任务，支持对模型认知能力的细致分析。通过在14个大型语言模型上进行广泛评估，包括近期增强推理能力的模型，发现模型在事实回忆任务与推理任务之间存在显著性能差距。MediQAl为评估法语医学问答中的语言模型表现提供了全面基准，填补了多语言医学资源的重要空白。
原文摘要 · Abstract (English)
This work introduces MediQAl, a French medical question answering dataset designed to evaluate the capabilities of language models in factual medical recall and reasoning over real-world clinical scenarios. MediQAl contains 32,603 questions sourced from French medical examinations across 41 medical subjects. The dataset includes three tasks: (i) Multiple-Choice Question with Unique answer, (ii) Multiple-Choice Question with Multiple answer, and (iii) Open-Ended Question with Short-Answer. Each question is labeled as Understanding or Reasoning, enabling a detailed analysis of models' cognitive capabilities. We validate the MediQAl dataset through extensive evaluation with 14 large language models, including recent reasoning-augmented models, and observe a significant performance gap between factual recall and reasoning tasks. Our evaluation provides a comprehensive benchmark for assessing language models' performance on French medical question answering, addressing a crucial gap in multilingual resources for the medical domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。