用检索增强多任务学习,从临床记录中高效提取膳食补充剂信息
RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplements
- 引入检索增强机制,通过相似样本召回提升模型泛化能力
- 在4类任务上均取得显著提升,最高达14.26%的性能增益
- 适合医疗文本挖掘、药物信息抽取等场景的研究者使用
本研究旨在构建一个先进的多任务大语言模型框架,从临床记录中提取膳食补充剂(DS)的多种信息。采用四项核心任务:命名实体识别(NER,2,949条句子)、关系抽取(RE,4,892条句子)、三元组抽取(TE,2,949条句子)和使用分类(UC,2,460条句子)。提出新型检索增强多任务信息抽取框架RAMIE,包含:1)使用任务特定提示进行指令微调;2)通过多任务训练提升存储效率并降低训练成本;3)引入检索增强生成(RAG),从训练集中检索相似示例。与仅使用指令微调的LLM对比,并开展消融实验评估多任务学习与RAG的贡献。结果显示,借助RAMIE框架,Llama2-13B在NER任务上达到87.39的F1分数(提升3.51%),在RE任务上达93.74(提升1.15%);Llama2-7B在TE任务上得79.45(提升14.26%),MedAlpaca-7B在UC任务上最高达93.45(提升0.94%)。消融实验表明,虽然多任务学习略有性能损失但提升效率,而RAG显著提高整体准确率。
原文摘要 · Abstract (English)
\textbf{Objective:} We aimed to develop an advanced multi-task large language model (LLM) framework to extract multiple types of information about dietary supplements (DS) from clinical records. \textbf{Methods:} We used four core DS information extraction tasks - namely, named entity recognition (NER: 2,949 clinical sentences), relation extraction (RE: 4,892 sentences), triple extraction (TE: 2,949 sentences), and usage classification (UC: 2,460 sentences) as our multitasks. We introduced a novel Retrieval-Augmented Multi-task Information Extraction (RAMIE) Framework, including: 1) employed instruction fine-tuning techniques with task-specific prompts, 2) trained LLMs for multiple tasks with improved storage efficiency and lower training costs, and 3) incorporated retrieval augmentation generation (RAG) techniques by retrieving similar examples from the training set. We compared RAMIE's performance to LLMs with instruction fine-tuning alone and conducted an ablation study to assess the contributions of multi-task learning and RAG to improved multitasking performance. \textbf{Results:} With the aid of the RAMIE framework, Llama2-13B achieved an F1 score of 87.39 (3.51\% improvement) on the NER task and demonstrated outstanding performance on the RE task with an F1 score of 93.74 (1.15\% improvement). For the TE task, Llama2-7B scored 79.45 (14.26\% improvement), and MedAlpaca-7B achieved the highest F1 score of 93.45 (0.94\% improvement) on the UC task. The ablation study revealed that while MTL increased efficiency with a slight trade-off in performance, RAG significantly boosted overall accuracy. \textbf{Conclusion:} This study presents a novel RAMIE framework that demonstrates substantial improvements in multi-task information extraction for DS-related data from clinical records. Our framework can potentially be applied to other domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。