用微调小模型私密问答病历，效果超GPT-4
Question Answering on Patient Medical Records with Private Fine-Tuned LLMs
- 先选相关病历资源，再基于选定内容回答问题
- 小模型（250倍更小）在任务1上F1高0.55%，任务2上Meteor高42%
- 适合注重隐私的医疗问答场景，支持本地部署
医疗系统持续生成大量电子健康记录（EHR），通常以快速医疗互操作性资源（FHIR）标准存储。尽管数据丰富，其复杂性和体量使用户难以提取关键健康信息。大语言模型（LLMs）可实现对医疗数据的语义问答，提升交互效率。但隐私与合规要求模型需在本地或私有环境部署。本文提出一种新方法：首先识别用户查询最相关的FHIR资源（任务1），再基于这些资源回答问题（任务2）。评估了私有部署的微调LLM性能，对比基准模型如GPT-4和GPT-4o。结果表明，微调模型虽仅为前者的1/250大小，在任务1的F1分数上领先0.55%，任务2的Meteor得分高出42%。同时研究了顺序微调、模型自评估及训练数据量对性能的影响。模型与数据集已公开于Hugging Face。
原文摘要 · Abstract (English)
Healthcare systems continuously generate vast amounts of electronic health records (EHRs), commonly stored in the Fast Healthcare Interoperability Resources (FHIR) standard. Despite the wealth of information in these records, their complexity and volume make it difficult for users to retrieve and interpret crucial health insights. Recent advances in Large Language Models (LLMs) offer a solution, enabling semantic question answering (QA) over medical data, allowing users to interact with their health records more effectively. However, ensuring privacy and compliance requires edge and private deployments of LLMs. This paper proposes a novel approach to semantic QA over EHRs by first identifying the most relevant FHIR resources for a user query (Task1) and subsequently answering the query based on these resources (Task2). We explore the performance of privately hosted, fine-tuned LLMs, evaluating them against benchmark models such as GPT-4 and GPT-4o. Our results demonstrate that fine-tuned LLMs, while 250x smaller in size, outperform GPT-4 family models by 0.55% in F1 score on Task1 and 42% on Meteor Task in Task2. Additionally, we examine advanced aspects of LLM usage, including sequential fine-tuning, model self-evaluation (narcissistic evaluation), and the impact of training data size on performance. The models and datasets are available here: https://huggingface.co/genloop
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。