用大模型提升科技文献中的技术项提取准确率
RATE: An LLM-Powered Retrieval Augmented Generation Technology-Extraction Pipeline
- 结合检索增强生成与多定义验证，提升候选技术项的召回与精度
- 在678篇脑机接口与扩展现实论文上实现91.27%的F1分数
- 适合科技情报分析、研究趋势洞察等场景的学者和机构使用
在技术快速变革的时代，技术图谱对决策至关重要，其构建依赖自动化技术提取方法。本文提出基于大语言模型的检索增强技术提取（RATE）管道，融合检索增强生成（RAG）与多定义驱动的LLM验证机制。该方法在候选生成中实现高召回，在筛选阶段保持高精度。尽管设计为通用流程，本文以678篇聚焦脑机接口（BCI）与扩展现实（XR）的研究文章为例进行验证。经验证的技术术语被映射为共现网络，揭示了研究领域的主题聚类与结构特征。为评估效果，专家人工标注了70篇随机选取文章的技术金标准数据集；另以双向编码器表示（BERT）模型作为对比。RATE达到91.27% F1分数，显著优于BERT的53.73%。结果表明，基于定义驱动的LLM方法在技术提取与图谱构建方面具有巨大潜力，并为BCI-XR领域新兴趋势提供了新洞见。源代码已公开于https://github.com/AryaAftab/RATE。
原文摘要 · Abstract (English)
In an era of radical technology transformations, technology maps play a crucial role in enhancing decision making. These maps heavily rely on automated methods of technology extraction. This paper introduces Retrieval Augmented Technology Extraction (RATE), a Large Language Model (LLM) based pipeline for automated technology extraction from scientific literature. RATE combines Retrieval Augmented Generation (RAG) with multi-definition LLM-based validation. This hybrid method results in high recall in candidate generation alongside with high precision in candidate filtering. While the pipeline is designed to be general and widely applicable, we demonstrate its use on 678 research articles focused on Brain-Computer Interfaces (BCIs) and Extended Reality (XR) as a case study. Consequently, The validated technology terms by RATE were mapped into a co-occurrence network, revealing thematic clusters and structural features of the research landscape. For the purpose of evaluation, a gold standard dataset of technologies in 70 selected random articles had been curated by the experts. In addition, a technology extraction model based on Bidirectional Encoder Representations of Transformers (BERT) was used as a comparative method. RATE achieved F1-score of 91.27%, Significantly outperforming BERT with F1-score of 53.73%. Our findings highlight the promise of definition-driven LLM methods for technology extraction and mapping. They also offer new insights into emerging trends within the BCI-XR field. The source code is available https://github.com/AryaAftab/RATE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。