arXiv:2409.03905cs.CL2024-09中稿 · publication in JAM…被引 4

构建癌症事件与关系的细粒度标注数据集,提升医疗文本信息抽取性能。

CACER: Clinical Concept Annotations for Cancer Events and Relations

  • 构建包含4.8万条医学事件和1万条药物-症状关系的标注数据集CACER。
  • 微调后的BERT和Llama3模型在事件抽取中达到88.2-88.0 F1,接近人工标注一致性(88.4 F1)。
  • 小模型优于大模型,证明高质量标注数据比大模型更关键,适合医疗NLP研究者使用。

临床笔记包含患者病史的非结构化描述,包括疾病与用药之间的关系。为研究抗癌药物与其相关症状负担的关系,我们从肿瘤科临床记录中提取医学问题和药物信息的结构化语义表示。本文提出临床概念标注数据集CACER,包含超过48,000个医学问题与药物事件,以及10,000条药物-问题和问题-问题关系的细粒度标注。基于CACER,我们开发并评估了BERT、Flan-T5、Llama3和GPT-4等基于Transformer的信息抽取模型,采用微调和上下文学习(ICL)方法。在事件抽取任务中,微调后的BERT和Llama3模型表现最佳,F1值达88.2-88.0,接近人工标注者间一致率(IAA)的88.4 F1。在关系抽取中,微调后的BERT、Flan-T5和Llama3表现最优,F1值为61.8-65.3。而采用ICL的GPT-4在两项任务中表现最差。微调模型显著优于GPT-4的ICL,凸显了标注训练数据与模型优化的重要性。此外,BERT模型性能与Llama3相当,表明在此任务中大型语言模型并无优势。结果强调了高质量标注数据对模型优化的关键作用。多个微调后的Transformer模型在多个抽取任务上达到了与人工标注一致率相当的水平。

原文摘要 · Abstract (English)

Clinical notes contain unstructured representations of patient histories, including the relationships between medical problems and prescription drugs. To investigate the relationship between cancer drugs and their associated symptom burden, we extract structured, semantic representations of medical problem and drug information from the clinical narratives of oncology notes. We present Clinical Concept Annotations for Cancer Events and Relations (CACER), a novel corpus with fine-grained annotations for over 48,000 medical problems and drug events and 10,000 drug-problem and problem-problem relations. Leveraging CACER, we develop and evaluate transformer-based information extraction (IE) models such as BERT, Flan-T5, Llama3, and GPT-4 using fine-tuning and in-context learning (ICL). In event extraction, the fine-tuned BERT and Llama3 models achieved the highest performance at 88.2-88.0 F1, which is comparable to the inter-annotator agreement (IAA) of 88.4 F1. In relation extraction, the fine-tuned BERT, Flan-T5, and Llama3 achieved the highest performance at 61.8-65.3 F1. GPT-4 with ICL achieved the worst performance across both tasks. The fine-tuned models significantly outperformed GPT-4 in ICL, highlighting the importance of annotated training data and model optimization. Furthermore, the BERT models performed similarly to Llama3. For our task, LLMs offer no performance advantage over the smaller BERT models. The results emphasize the need for annotated training data to optimize models. Multiple fine-tuned transformer models achieved performance comparable to IAA for several extraction tasks.

医疗NLP信息抽取细粒度标注BERT

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。