用大模型生成疾病术语,显著提升医学实体归一化准确率。
Disease Entity Recognition and Normalization is Improved with Large Language Model Derived Synthetic Normalized Mentions
- 用LLaMa-2生成统一医学语言系统中的疾病标准化术语作为训练数据。
- 实体归一化在整体和分布外数据上提升20-55个百分点,命名识别小幅提升1-2点。
- 适合医疗信息提取、疾病知识图谱构建等需要高精度归一化的研究者。
临床命名实体识别与实体归一化系统通常依赖标注语料库和知识图谱(KG)进行学习。然而,罕见概念在训练语料中提及少,且在大型知识图谱中也缺乏详细描述或同义词。针对疾病实体识别(DER)与疾病实体归一化(DEN),这导致高质量训练样本远少于已知疾病数量。利用大语言模型(LLM)生成合成训练样本可改善信息抽取性能。我们微调了LLaMa-2 13B聊天模型,生成包含统一医学语言系统(UMLS)疾病语义组概念的标准化术语合成语料。评估了三种疾病语料库上四种数据增强策略下,使用BioBERT(DER)、SapBERT与KrissBERT(DEN)的总体及分布外(OOD)性能表现。结果表明,合成数据使两种模型在所有三组数据上的总体精度提升3-9个百分点,分布外性能提升20-55个百分点;命名识别总体性能仅提升1-2点,且仅一个数据集显示分布外改进。消融实验表明,性能提升主要源于归一化任务本身,而非仅分布外泛化能力。该方法对归一化效果明显,对识别帮助有限。代码与数据集已公开。
原文摘要 · Abstract (English)
Background: Machine learning methods for clinical named entity recognition and entity normalization systems can utilize both labeled corpora and Knowledge Graphs (KGs) for learning. However, infrequently occurring concepts may have few mentions in training corpora and lack detailed descriptions or synonyms, even in large KGs. For Disease Entity Recognition (DER) and Disease Entity Normalization (DEN), this can result in fewer high quality training examples relative to the number of known diseases. Large Language Model (LLM) generation of synthetic training examples could improve performance in these information extraction tasks. Methods: We fine-tuned a LLaMa-2 13B Chat LLM to generate a synthetic corpus containing normalized mentions of concepts from the Unified Medical Language System (UMLS) Disease Semantic Group. We measured overall and Out of Distribution (OOD) performance for DER and DEN, with and without synthetic data augmentation. We evaluated performance on 3 different disease corpora using 4 different data augmentation strategies, assessed using BioBERT for DER and SapBERT and KrissBERT for DEN. Results: Our synthetic data yielded a substantial improvement for DEN, in all 3 training corpora the top 1 accuracy of both SapBERT and KrissBERT improved by 3-9 points in overall performance and by 20-55 points in OOD data. A small improvement (1-2 points) was also seen for DER in overall performance, but only one dataset showed OOD improvement. Conclusion: LLM generation of normalized disease mentions can improve DEN relative to normalization approaches that do not utilize LLMs to augment data with synthetic mentions. Ablation studies indicate that performance gains for DEN were only partially attributable to improvements in OOD performance. The same approach has only a limited ability to improve DER. We make our software and dataset publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。