用医学知识图谱提升大模型,比纯文本预训练更准且可更新。
Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG
- 用UMLS构建342万关系的知识图谱,持续预训练模型注入知识
- 在医学问答任务上,新方法比原模型准确率提升超3点,多跳推理更强
- 无需重训练即可增强大模型,适合需要动态更新医学知识的场景
将领域知识注入语言模型对生物医学应用至关重要。本文探索两种互补策略:一是基于UMLS Metathesaurus构建包含342万条关系的大型生物医学知识图谱(存储于Neo4j),并从中提取约1亿词元文本用于持续预训练,得到BERTUMLS与BioBERTUMLS;二是提出GraphRAG,在推理时查询知识图谱。在六个BLURB基准任务上,BERTUMLS在知识密集型问答中表现最佳;而以已具备丰富生物医学知识的BioBERT为基础的模型增益较弱,表明存在边际递减效应。此外,将LLaMA 3-8B与GraphRAG结合后,在PubMedQA上准确率提升超3点,BioASQ上提升5点,无需任何重训练,实现透明、多跳、易更新的知识访问。研究开源了处理后的UMLS Neo4j图谱,支持复现。
原文摘要 · Abstract (English)
The injection of domain-specific knowledge is crucial for adapting language models (LMs) to specialized fields such as biomedicine. While most current approaches rely on unstructured text corpora, this study explores two complementary strategies for leveraging structured knowledge from the UMLS Metathesaurus: (i) Continual pretraining that embeds knowledge into model parameters, and (ii) Graph Retrieval-Augmented Generation (GraphRAG) that consults a knowledge graph at inference time. We first construct a large-scale biomedical knowledge graph from UMLS (3.4 million concepts and 34.2 million relations), stored in Neo4j for efficient querying. We then derive a ~100-million-token textual corpus from this graph to continually pretrain two models: BERTUMLS (from BERT) and BioBERTUMLS (from BioBERT). We evaluate these models on six BLURB (Biomedical Language Understanding and Reasoning Benchmark) datasets spanning five task types and evaluate GraphRAG on the two QA (Question Answering) datasets (PubMedQA, BioASQ). On BLURB tasks, BERTUMLS improves over BERT, with the largest gains on knowledge-intensive QA. Effects on BioBERT are more nuanced, suggesting diminishing returns when the base model already encodes substantial biomedical text knowledge. Finally, augmenting LLaMA 3-8B with our GraphRAG pipeline yields over than 3 points accuracy on PubMedQA and 5 points on BioASQ without any retraining, delivering transparent, multi-hop, and easily updated knowledge access. We release the processed UMLS Neo4j graph to support reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。