用开源大模型自动提取病历中的系统回顾信息,省成本还准。
A Large Language Model Based Pipeline for Review of Systems Entity Recognition from Clinical Notes
- 用大模型分步识别症状、体征及所属系统,支持零样本和少样本
- 最大模型达F1 0.952,小模型仅用三分之一显存也表现良好
- 新归因算法提升匹配精度,适合医疗数据量小的场景
目标:开发一种基于大语言模型(LLM)的低成本自动化流程,从临床笔记中提取系统回顾(ROS)实体。方法:首先利用SecTag头术语定位ROS章节,再通过少量样本的LLM识别疾病或症状、其阳性/阴性状态及其所属身体系统。采用4个开源LLM模型:llama3.1:8b、gemma3:27b、mistral3.1:24b和gpt-oss:20b。引入一种新型归因算法,解决非精确匹配与同义词匹配问题。在包含340个标注实体的24份全科医学病历上进行评估。结果:开源LLM实现本地部署且成本低,性能良好。Gemma、Mistral和Gpt-oss在实体识别、否定检测和系统分类三项任务中表现稳健,最高F1得分为0.952。引入归因算法后,所有模型在F1、准确率上均有提升,错误率下降。值得注意的是,较小的Llama模型虽仅使用其他模型约三分之一的显存,仍取得良好效果。讨论与结论:该流程可规模化部署于本地,减轻医生记录负担;开源大模型为资源有限的医疗机构提供实用的AI选择。方法上,新归因算法有效提升零/少样本场景下命名实体识别的准确性。
原文摘要 · Abstract (English)
Objective: Develop a cost-effective, large language model (LLM)-based pipeline for automatically extracting Review of Systems (ROS) entities from clinical notes. Materials and Methods: The pipeline extracts ROS section from the clinical note using SecTag header terminology, followed by few-shot LLMs to identify ROS entities such as diseases or symptoms, their positive/negative status and associated body systems. We implemented the pipeline using 4 open-source LLM models: llama3.1:8b, gemma3:27b, mistral3.1:24b and gpt-oss:20b. Additionally, we introduced a novel attribution algorithm that aligns LLM-identified ROS entities with their source text, addressing non-exact and synonymous matches. The evaluation was conducted on 24 general medicine notes containing 340 annotated ROS entities. Results: Open-source LLMs enable a local, cost-efficient pipeline while delivering promising performance. Larger models like Gemma, Mistral, and Gpt-oss demonstrate robust performance across three entity recognition tasks of the pipeline: ROS entity extraction, negation detection and body system classification (highest F1 score = 0.952). With the attribution algorithm, all models show improvements across key performance metrics, including higher F1 score and accuracy, along with lower error rate. Notably, the smaller Llama model also achieved promising results despite using only one-third the VRAM of larger models. Discussion and Conclusion: From an application perspective, our pipeline provides a scalable, locally deployable solution to easing the ROS documentation burden. Open-source LLMs offer a practical AI option for resource-limited healthcare settings. Methodologically, our newly developed algorithm facilitates accuracy improvements for zero- and few-shot LLMs in named entity recognition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。