用知识图谱增强医学语言模型，提升对专业概念的理解。
BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment
- 联合预训练语言模型与知识图谱，通过子图匹配实现跨模态对齐。
- 在PubMedBERT等模型上提升实体表示质量，小数据集即可见效。
- 适合医学文本理解、生物信息检索等需要精准概念推理的场景。
近年来，预训练语言模型在改善生物医学文本理解方面取得显著进展。然而，现有生物医学大模型对复杂领域概念结构和知识图谱中编码的事实信息仍理解有限。本文提出BALI（Biomedical Knowledge Graph and Language Model Alignment），一种新的语言模型与知识图谱联合预训练方法，通过同时学习专用知识图谱编码器并对齐语言模型与图谱的表征，将外部知识注入语言模型。对于给定文本序列，将生物医学概念提及链接至统一医学语言系统（UMLS）知识图谱，并利用局部子图作为这些提及的跨模态正样本。实证结果表明，在PubMedBERT和BioLinkBERT等主流生物医学语言模型上应用该方法，可在多种语言理解任务中提升性能，并改善实体表征质量，即使仅使用来自科学摘要的小规模对齐数据进行少量预训练亦有效。
原文摘要 · Abstract (English)
In recent years, there has been substantial progress in using pretrained Language Models (LMs) on a range of tasks aimed at improving the understanding of biomedical texts. Nonetheless, existing biomedical LLMs show limited comprehension of complex, domain-specific concept structures and the factual information encoded in biomedical Knowledge Graphs (KGs). In this work, we propose BALI (Biomedical Knowledge Graph and Language Model Alignment), a novel joint LM and KG pre-training method that augments an LM with external knowledge by the simultaneous learning of a dedicated KG encoder and aligning the representations of both the LM and the graph. For a given textual sequence, we link biomedical concept mentions to the Unified Medical Language System (UMLS) KG and utilize local KG subgraphs as cross-modal positive samples for these mentions. Our empirical findings indicate that implementing our method on several leading biomedical LMs, such as PubMedBERT and BioLinkBERT, improves their performance on a range of language understanding tasks and the quality of entity representations, even with minimal pre-training on a small alignment dataset sourced from PubMed scientific abstracts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。