医学大模型的专科数据真能提升表现吗？研究发现未必。
What Does Neuro Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs
- 构建了24000+样本的跨专科医学问答数据集S-MedQA
- 专科微调不必然提升对应领域性能，通用医学迁移更关键
- 适合关注医疗AI知识注入与微调策略的研究者
本文提出S-MedQA，一个用于评估大语言模型在细粒度临床专科中表现的英文医学问答基准数据集，包含超过24,000个样本，覆盖15个医学专科，支持多专科标注。数据通过机器与专家双重验证，确保可靠性。我们利用该数据集研究临床专科数据在医学问答任务中的作用。结果表明，基于某专科数据训练的模型并不一定在该专科上表现最优；无论模型在哪个专科微调，所有专科的临床相关术语概率均持续上升。由此推测，在当前设定下，性能提升主要源于领域迁移（如通用到医学），而非引入专科特异性知识。这一发现提示需重新思考医学大模型微调数据的作用。为推动临床NLP发展，我们开源S-MedQA及全部实验代码。
原文摘要 · Abstract (English)
In this paper, we introduce S-MedQA, an English medical question-answering (QA) dataset designed for benchmarking large language models (LLMs) in fine-grained clinical specialties. S-MedQA consists of over 24k examples, covering 15 medical specialties, with QA pairs that can have multiple specialty annotations, such as when a question is cross-disciplinary. The dataset is constructed using both machine and expert verification to maximize data availability and reliability. We use S-MedQA to investigate the role of clinical specialties in the knowledge-intensive scenario of medical QA. Our results show that training on data from a clinical specialty does not necessarily lead to the best performance on that specialty. Additionally, regardless of the specialty the LLM was fine-tuned on, token probabilities of clinically relevant terms consistently increase across all specialties. Based on these findings, we hypothesize that improvement gains, at least in our settings, are derived primarily from domain shifting (e.g., general to medical) rather than from injecting specialty-specific knowledge. This suggests a need to rethink the role of fine-tuning data in the medical domain. To encourage further advancements in the clinical NLP field, we release S-MedQA along with all the code required to reproduce our experiments for the research community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。