用大模型自动标注实体识别数据，效果接近人工。
LLMs as Data Annotators: How Close Are We to Human Performance
- 用检索增强生成自动选上下文样例，替代手动挑选。
- 70亿参数模型在多个数据集上表现接近人类标注水平。
- 小模型+好检索比大模型更高效，适合资源有限场景。
在自然语言处理中，微调大模型在各类应用中表现良好，但依赖高质量标注数据。然而，人工标注成本高、耗时长。因此，越来越多研究使用大模型自动化标注，通常采用上下文学习（ICL），即在提示中加入任务相关示例以提升性能。但手动选择上下文示例效率低且影响效果。本文针对命名实体识别（NER）任务，在多个数据集上系统比较了不同大模型（约7B与70B参数）、多种嵌入模型的表现。实验涵盖专有与开源模型，并借鉴检索增强生成（RAG）思想，提出自动检索上下文样例的方法，克服ICL局限性。结果表明：选择合适的模型与嵌入方式至关重要；模型规模与性能间存在权衡；未来研究应聚焦更具挑战性的数据集。
原文摘要 · Abstract (English)
In NLP, fine-tuning LLMs is effective for various applications but requires high-quality annotated data. However, manual annotation of data is labor-intensive, time-consuming, and costly. Therefore, LLMs are increasingly used to automate the process, often employing in-context learning (ICL) in which some examples related to the task are given in the prompt for better performance. However, manually selecting context examples can lead to inefficiencies and suboptimal model performance. This paper presents comprehensive experiments comparing several LLMs, considering different embedding models, across various datasets for the Named Entity Recognition (NER) task. The evaluation encompasses models with approximately $7$B and $70$B parameters, including both proprietary and non-proprietary models. Furthermore, leveraging the success of Retrieval-Augmented Generation (RAG), it also considers a method that addresses the limitations of ICL by automatically retrieving contextual examples, thereby enhancing performance. The results highlight the importance of selecting the appropriate LLM and embedding model, understanding the trade-offs between LLM sizes and desired performance, and the necessity to direct research efforts towards more challenging datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。