arXiv:2504.00664cs.CL2025-04中稿 · appear in IEEE ICH…被引 11

LLMs在生物医学命名实体识别上表现更优,但代价是计算成本极高。

Do LLMs Surpass Encoders for Biomedical NER?

  • 用BIO标注法保留位置信息,对比编码器与解码器模型的实体识别效果。
  • 在五组数据上,LLMs比BERT类模型高2-8% F1分数,长实体优势更明显。
  • 适合对精度要求高且不介意算力开销的研究场景,实时应用仍推荐编码器。

在自由文本中识别生物医学概念及其类型(如药物或基因)的命名实体识别(NER)是信息抽取的核心组件。当前主流方法从传统机器学习转向基于Transformer的编码器模型(如BERT),但解码器模型(即大语言模型,LLMs)在信息抽取中日益流行。然而,由于生成式特性,LLMs常忽略位置信息,且推理成本极高。本文采用统一的BIO标注方案,在五个不同数据集上比较了同为80亿参数级别的Mistral和Llama等LLMs与BERT、BiomedBERT、DeBERTa-v3(约3亿参数)等编码器模型的表现。结果显示,除一个数据集外,LLMs在F1分数上普遍领先2%-8%,尤其在长度≥3个词元的长实体上优势显著。但其推理时间高出一到两个数量级,硬件需求也相应提升。因此,当性能差距小或需要实时响应时,编码器模型仍具优势。

原文摘要 · Abstract (English)

Recognizing spans of biomedical concepts and their types (e.g., drug or gene) in free text, often called biomedical named entity recognition (NER), is a basic component of information extraction (IE) pipelines. Without a strong NER component, other applications, such as knowledge discovery and information retrieval, are not practical. State-of-the-art in NER shifted from traditional ML models to deep neural networks with transformer-based encoder models (e.g., BERT) emerging as the current standard. However, decoder models (also called large language models or LLMs) are gaining traction in IE. But LLM-driven NER often ignores positional information due to the generative nature of decoder models. Furthermore, they are computationally very expensive (both in inference time and hardware needs). Hence, it is worth exploring if they actually excel at biomedical NER and assess any associated trade-offs (performance vs efficiency). This is exactly what we do in this effort employing the same BIO entity tagging scheme (that retains positional information) using five different datasets with varying proportions of longer entities. Our results show that the LLMs chosen (Mistral and Llama: 8B range) often outperform best encoder models (BERT-(un)cased, BiomedBERT, and DeBERTav3: 300M range) by 2-8% in F-scores except for one dataset, where they equal encoder performance. This gain is more prominent among longer entities of length >= 3 tokens. However, LLMs are one to two orders of magnitude more expensive at inference time and may need cost prohibitive hardware. Thus, when performance differences are small or real time user feedback is needed, encoder models might still be more suitable than LLMs.

生物医学NER大模型实体识别效率对比

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。