arXiv:2502.16022cs.CL2025-02被引 1

用数据增强提升大模型识别病历术语能力,助力患者理解医疗记录。

Enhancing LLMs for Identifying and Prioritizing Important Medical Jargons from Electronic Health Record Notes Utilizing Data Augmentation: A Comparative Study

  • 结合提示工程与GPT-4o生成数据,增强小样本下术语提取效果。
  • 开源模型在精调后表现最优,尤其在Jaccard匹配下F1达0.431。
  • 数据增强效果优于微调,但质量依赖生成数据的准确性。

OpenNotes让患者可访问电子健康记录(EHR)笔记,但密集的医学术语限制了理解。本文评估闭源与开源大模型在提取并优先排序对个体患者最相关的术语方面的能力,使用90份专家标注的EHR笔记。测试了通用与结构化提示、零样本与少样本提示、微调以及基于GPT-4o的数据增强(配以排序技术优化低资源场景训练)。通过将数据集规模从10扩展至9,995例进行微调,所有设置均采用10折交叉验证,报告在宽松匹配和杰卡德指数两种字符串匹配标准下的F1与平均倒数排名(MRR),并进行输出错误分析。结果显示:在金标准数据集上微调后,开源模型表现最佳——在杰卡德匹配下,DeepSeek 8B达到最高F1(0.431,SD 0.046),BioMistral 7B达到最高MRR(0.577,SD 0.109)。但在宽松匹配下,即使经增强或微调,开源模型仍不及闭源模型。少样本提示未优于零样本;提示风格显著影响结果;在小规模金标准集上微调可提升性能;数据增强效果可媲美甚至超越微调,但其收益高度依赖生成数据质量。研究证明,提示策略、微调与数据增强均能有效提升大模型在低资源临床场景中面向患者的术语提取性能。

原文摘要 · Abstract (English)

OpenNotes gives patients access to their EHR notes, but dense medical jargon limits comprehension. We evaluate closed-source and open-source LLMs for extracting and prioritizing the jargon terms most relevant to individual patients, using 90 expert-annotated EHR notes. We test combinations of general vs. structured prompts, zero-shot vs. few-shot prompting, fine-tuning, and GPT-4o-based data augmentation, the last paired with a ranking technique to refine training in low-resource settings. To assess the effect of dataset size, we fine-tune on augmented datasets scaled from 10 to 9,995 examples. All settings are evaluated with 10-fold cross-validation, reporting F1 and Mean Reciprocal Rank (MRR) under two string-matching criteria (relaxed matching and Jaccard Index), followed by an error analysis of model outputs. Open-source models performed best when fine-tuned on the gold-standard dataset: under Jaccard-based matching, DeepSeek 8B achieved the top F1 (0.431, SD 0.046) and BioMistral 7B the top MRR (0.577, SD 0.109). Under relaxed matching, however, open-source models did not match closed-source performance even with augmentation or fine-tuning. Few-shot prompting offered no advantage over zero-shot in vanilla models; prompting style substantially affected results; fine-tuning on a small gold-standard set improved performance; and data augmentation matched or exceeded fine-tuning, though its benefit depended heavily on augmented-data quality. These findings show that prompting strategy, fine-tuning, and data augmentation each meaningfully improve LLM performance on patient-centered jargon extraction in low-resource clinical settings.

大模型医疗文本数据增强术语提取

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。