对比大模型与小模型在危重症患者休克预测中的表现,发现大模型无明显优势。
Benchmarking LLMs for Predictive Applications in the Intensive Care Units
- 用临床数据微调GatorTron、Llama等大模型,对比其预测休克能力
- GatorTron基线模型召回率达80.5%,但大模型与小模型整体性能相近
- 提示未来应聚焦临床轨迹预测,而非简单命名实体识别
随着大语言模型(LLM)的发展,自然语言处理领域的诸多任务已发生变革。然而,其在预测性任务中的应用仍研究不足。本研究对比了GatorTron-Base(基于临床数据训练)、Llama 8B、Mistral 7B等大模型,以及BioBERT、DocBERT、BioClinicalBERT、Word2Vec和Doc2Vec等模型在预测危重症患者休克方面的表现。基于MIMIC-III数据库中17,294例患者的文本数据,筛选出住院时长>24小时且休克指数(SI)>0.7的患者,共得355例正常与87例异常患者。微调时采用焦点损失与交叉熵损失以缓解类别不平衡问题。结果表明,尽管GatorTron Base达到最高加权召回率80.5%,但大模型与小模型的整体性能差异不大。这说明,尽管大模型在文本任务上表现优异,但在预测临床事件方面并无本质优势。为实现真正临床价值,未来应重点发展能预测临床轨迹的大模型,而非局限于命名实体识别或表型标注等简单任务。
原文摘要 · Abstract (English)
With the advent of LLMs, various tasks across the natural language processing domain have been transformed. However, their application in predictive tasks remains less researched. This study compares large language models, including GatorTron-Base (trained on clinical data), Llama 8B, and Mistral 7B, against models like BioBERT, DocBERT, BioClinicalBERT, Word2Vec, and Doc2Vec, setting benchmarks for predicting Shock in critically ill patients. Timely prediction of shock can enable early interventions, thus improving patient outcomes. Text data from 17,294 ICU stays of patients in the MIMIC III database were scored for length of stay > 24 hours and shock index (SI) > 0.7 to yield 355 and 87 patients with normal and abnormal SI-index, respectively. Both focal and cross-entropy losses were used during finetuning to address class imbalances. Our findings indicate that while GatorTron Base achieved the highest weighted recall of 80.5%, the overall performance metrics were comparable between SLMs and LLMs. This suggests that LLMs are not inherently superior to SLMs in predicting future clinical events despite their strong performance on text-based tasks. To achieve meaningful clinical outcomes, future efforts in training LLMs should prioritize developing models capable of predicting clinical trajectories rather than focusing on simpler tasks such as named entity recognition or phenotyping.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。