大模型可高效自动分类科学文献，准确率达82%。
On the Effectiveness of Large Language Models in Automating Categorization of Scientific Texts
- 用大模型对科学文本进行分级分类，替代传统方法。
- 最新模型准确率82%，比BERT高8个百分点。
- 适合需要处理海量论文的科研机构与学术平台。
大语言模型（LLMs）的快速发展带来了众多应用机会。信息检索系统中传统的文本摘要与分类任务，在应对科学出版物等大量文献时尤为重要。随着科学知识迅速增长，研究致力于构建新型研究信息系统，不仅支持关键词搜索，还能自动识别高校与产业界的知识组织中的研究领域。为实现这一目标，我们评估了多种大模型在将科学文献归入层级分类体系中的表现。以FORC数据集为真实标签，发现近期大模型（如Meta Llama 3.1）最高可达0.82的准确率，较传统BERT模型提升0.08。
原文摘要 · Abstract (English)
The rapid advancement of Large Language Models (LLMs) has led to a multitude of application opportunities. One traditional task for Information Retrieval systems is the summarization and classification of texts, both of which are important for supporting humans in navigating large literature bodies as they e.g. exist with scientific publications. Due to this rapidly growing body of scientific knowledge, recent research has been aiming at building research information systems that not only offer traditional keyword search capabilities, but also novel features such as the automatic detection of research areas that are present at knowledge intensive organizations in academia and industry. To facilitate this idea, we present the results obtained from evaluating a variety of LLMs in their ability to sort scientific publications into hierarchical classifications systems. Using the FORC dataset as ground truth data, we have found that recent LLMs (such as Meta Llama 3.1) are able to reach an accuracy of up to 0.82, which is up to 0.08 better than traditional BERT models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。