用医学问答数据微调大模型,提升诊断辅助AI的准确率。
A Comprehensive Study on Fine-Tuning Large Language Models for Medical Question Answering Using Classification Models and Comparative Analysis
- 分两阶段:先分类问题再匹配答案,提升响应精准度。
- BERT Large Uncased 模型在测试中达到100%各项指标表现。
- 基于6800条真实数据+合成数据,适合医疗AI研发人员参考。
本文系统研究了专用于医学问答的大语言模型(LLMs)的开发与微调方法,旨在提升回答医学问题的准确性和效率。采用两阶段策略:首先对输入的医学问题进行特定标签预测,随后针对该标签生成预设答案。评估了RoBERTa、BERT等模型的性能,训练数据来源于从Healthline.com抓取的6,800个样本,并结合合成数据增强。通过5折交叉验证进行评估,使用准确率、精确率、召回率和F1分数作为指标,同时记录训练时间。结果显示,LoRA-RoBERTa-large模型达到78.47%准确率、72.91%精确率、76.95%召回率和73.56% F1分数;RoBERTa-base模型表现优异,准确率99.87%、精确率99.81%、召回率99.86%、F1分数99.82%;BERT Uncased模型准确率95.85%、精确率94.42%、召回率95.58%、F1分数94.72%;而BERT Large Uncased模型在所有指标上均达到100%。实验表明,这些模型具备强大的医学问题分类与精准答答回应能力,为改进健康领域AI解决方案提供支持。
原文摘要 · Abstract (English)
This paper presents the overview of the development and fine-tuning of large language models (LLMs) designed specifically for answering medical questions. We are mainly improving the accuracy and efficiency of providing reliable answers to medical queries. In our approach, we have two stages, prediction of a specific label for the received medical question and then providing a predefined answer for this label. Various models such as RoBERTa and BERT were examined and evaluated based on their ability. The models are trained using the datasets derived from 6,800 samples that were scraped from Healthline. com with additional synthetic data. For evaluation, we conducted a comparative study using 5-fold cross-validation. For accessing performance we used metrics like, accuracy, precision, recall, and F1 score and also recorded the training time. The performance of the models was evaluated using 5-fold cross-validation. The LoRA Roberta-large model achieved an accuracy of 78.47%, precision of 72.91%, recall of 76.95%, and an F1 score of 73.56%. The Roberta-base model demonstrated high performance with an accuracy of 99.87%, precision of 99.81%, recall of 99.86%, and an F1 score of 99.82%. The Bert Uncased model showed strong results with an accuracy of 95.85%, precision of 94.42%, recall of 95.58%, and an F1 score of 94.72%. Lastly, the Bert Large Uncased model achieved the highest performance, with an accuracy, precision, recall, and F1 score of 100%. The results obtained have helped indicate the capability of the models in classifying the medical questions and generating accurate answers in the prescription of improved health-related AI solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。