对比大模型与BERT在临床文本抽取中的表现，发现大模型更准但更慢。
Information Extraction from Clinical Notes: Are We Ready to Switch to Large Language Models?
- 用LLaMA和BERT对比临床实体与关系抽取，覆盖4类实体16种修饰词。
- 数据少时大模型提升明显，i2b2上大模型比BERT高7%（NER）和4%（RE）。
- 适合追求精度且有算力的场景，不适合高吞吐或资源受限应用。
背景：信息抽取（IE）在临床自然语言处理中至关重要。尽管大语言模型（LLMs）在生成任务中表现出色，其在抽取任务中的表现仍存争议。方法：我们基于四个来源（UT Physicians、MTSamples、MIMIC-III 和 i2b2）共1,588份临床病历，研究命名实体识别（NER）与关系抽取（RE）。构建了一个涵盖4类临床实体和16种修饰词的标注语料库，并将指令微调的LLaMA-2和LLaMA-3与BERT在性能、泛化能力、计算资源消耗及吞吐量方面进行对比。结果：在所有数据集上，LLaMA模型均优于BERT。在充足训练数据下，性能提升较小（NER提升1%，RE提升1.5%-3.7%）；在数据有限时提升更大。在未见的i2b2数据上，LLaMA-3-70B在NER上比BERT高出7%（F1），RE高出4%。然而，LLaMA模型需要更多计算资源，运行速度最高慢28倍。我们开发了名为“Kiwi”的临床信息抽取工具包，集成两种模型，已开源至https://kiwi.clinicalnlp.org/。结论：本研究是首个使用开源大模型构建并评估综合性临床信息抽取系统的实证工作。结果表明，大模型在临床NER和RE上优于BERT，但伴随更高的计算成本和更低的吞吐量。因此，在临床信息抽取中选择模型应结合任务需求，权衡性能与实际部署条件。
原文摘要 · Abstract (English)
Backgrounds: Information extraction (IE) is critical in clinical natural language processing (NLP). While large language models (LLMs) excel on generative tasks, their performance on extractive tasks remains debated. Methods: We investigated Named Entity Recognition (NER) and Relation Extraction (RE) using 1,588 clinical notes from four sources (UT Physicians, MTSamples, MIMIC-III, and i2b2). We developed an annotated corpus covering 4 clinical entities and 16 modifiers, and compared instruction-tuned LLaMA-2 and LLaMA-3 against BERT in terms of performance, generalizability, computational resources, and throughput to BERT. Results: LLaMA models outperformed BERT across datasets. With sufficient training data, LLaMA showed modest improvements (1% on NER, 1.5-3.7% on RE); improvements were larger with limited training data. On unseen i2b2 data, LLaMA-3-70B outperformed BERT by 7% (F1) on NER and 4% on RE. However, LLaMA models required more computing resources and ran up to 28 times slower. We implemented "Kiwi," a clinical IE package featuring both models, available at https://kiwi.clinicalnlp.org/. Conclusion: This study is among the first to develop and evaluate a comprehensive clinical IE system using open-source LLMs. Results indicate that LLaMA models outperform BERT for clinical NER and RE but with higher computational costs and lower throughputs. These findings highlight that choosing between LLMs and traditional deep learning methods for clinical IE applications should remain task-specific, taking into account both performance metrics and practical considerations such as available computing resources and the intended use case scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。