用提示工程提升大模型临床时间关系抽取能力，效果超当前最佳
Prompting Large Language Models for Clinical Temporal Relation Extraction
- 设计多种提示与微调策略，适配不同规模模型
- 全监督下最高达89.54%准确率，超越现有最佳3.74%
- 适合医疗文本分析、临床决策支持等场景研究者参考
本研究旨在探索在少样本和全监督条件下，利用大语言模型（LLM）进行临床时间关系抽取（CTRE）。实验采用四种LLM：基于编码器的GatorTron-Base（345M）/Large（8.9B）及基于解码器的LLaMA3-8B/MeLLaMA-13B。提出全量微调（FFT）与参数高效微调（PEFT）策略，并在2012年i2b2 CTRE任务上评估。对GatorTron-Base测试四种策略：标准微调、未冻结模型的硬提示、冻结模型的软提示、冻结模型的低秩适应（LoRA）。对GatorTron-Large采用软提示与LoRA结合量化技术。此外，LLaMA3-8B与MeLLaMA-13B使用量化低秩适应（QLoRA）与指令微调、标准微调。结果表明，在全监督设置下，未冻结的GatorTron-Base硬提示达到89.54%的F1分数，优于现有最佳模型（85.70%）3.74个百分点；两种QLoRA变体（应用于GatorTron-Large）及标准微调的GatorTron-Base分别提升2.36%、1.88%、0.25%。解码器模型在全监督中表现更优，但在少样本场景下反超。研究验证了提示与微调策略对提升CTRE性能的关键作用，强调应根据数据量与任务需求选择合适模型与方法。
原文摘要 · Abstract (English)
Objective: This paper aims to prompt large language models (LLMs) for clinical temporal relation extraction (CTRE) in both few-shot and fully supervised settings. Materials and Methods: This study utilizes four LLMs: Encoder-based GatorTron-Base (345M)/Large (8.9B); Decoder-based LLaMA3-8B/MeLLaMA-13B. We developed full (FFT) and parameter-efficient (PEFT) fine-tuning strategies and evaluated these strategies on the 2012 i2b2 CTRE task. We explored four fine-tuning strategies for GatorTron-Base: (1) Standard Fine-Tuning, (2) Hard-Prompting with Unfrozen LLMs, (3) Soft-Prompting with Frozen LLMs, and (4) Low-Rank Adaptation (LoRA) with Frozen LLMs. For GatorTron-Large, we assessed two PEFT strategies-Soft-Prompting and LoRA with Frozen LLMs-leveraging Quantization techniques. Additionally, LLaMA3-8B and MeLLaMA-13B employed two PEFT strategies: LoRA strategy with Quantization (QLoRA) applied to Frozen LLMs using instruction tuning and standard fine-tuning. Results: Under fully supervised settings, Hard-Prompting with Unfrozen GatorTron-Base achieved the highest F1 score (89.54%), surpassing the SOTA model (85.70%) by 3.74%. Additionally, two variants of QLoRA adapted to GatorTron-Large and Standard Fine-Tuning of GatorTron-Base exceeded the SOTA model by 2.36%, 1.88%, and 0.25%, respectively. Decoder-based models with frozen parameters outperformed their Encoder-based counterparts in this setting; however, the trend reversed in few-shot scenarios. Discussions and Conclusions: This study presented new methods that significantly improved CTRE performance, benefiting downstream tasks reliant on CTRE systems. The findings underscore the importance of selecting appropriate models and fine-tuning strategies based on task requirements and data availability. Future work will explore larger models and broader CTRE applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。