用大模型零样本提取生物医学关系，测试三款OpenAI模型表现
A Benchmark for End-to-End Zero-Shot Biomedical Relation Extraction with LLMs: Experiments with OpenAI Models
- 构建七个特色数据集，评估大模型端到端零样本关系抽取能力
- 在部分数据集上接近有监督方法性能，复杂多关系输入仍易出错
- 适合关注大模型在生物医学知识提取中应用的研究者参考
从科学文献中提取关系是生物医学自然语言处理的基础任务，因为实体及其关系驱动假设生成与知识发现。随着文献量激增，关系抽取（RE）对于构建可用于计算的结构化知识图谱至关重要。大模型兴起后，是否可跳过标注、直接通过调用API实现零样本关系抽取（ZSRE）成为关键问题。本文提出一个包含七个生物医学关系抽取数据集的基准，评估GPT-4、o1和GPT-OSS-120B三款OpenAI模型在端到端零样本关系抽取中的表现。结果表明，大模型在部分数据集上的表现已接近有监督方法，但在表达多个不同谓词关系的复杂输入上仍存在困难。错误分析揭示了改进空间。
原文摘要 · Abstract (English)
Extracting relations from scientific literature is a fundamental task in biomedical NLP because entities and relations among them drive hypothesis generation and knowledge discovery. As literature grows rapidly, relation extraction (RE) is indispensable to curate knowledge graphs to be used as computable structured and symbolic representations. With the rise of LLMs, it is pertinent to examine if it is better to skip tailoring supervised RE methods, save annotation burden, and just use zero shot RE (ZSRE) via LLM API calls. In this paper, we propose a benchmark with seven biomedical RE datasets with interesting characteristics and evaluate three Open AI models (GPT-4, o1, and GPT-OSS-120B) for end-to-end ZSRE. We show that LLM-based ZSRE is inching closer to supervised methods in performances on some datasets but still struggles on complex inputs expressing multiple relations with different predicates. Our error analysis reveals scope for improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。