首次系统评估大模型在专业领域数据标注中的表现。
Are Expert-Level Language Models Expert-Level Annotators?
- 在三个高专业领域测试大模型标注能力
- 发现其表现优于普通人类标注员但仍有局限
- 适合成本敏感的专家级标注场景
数据标注指对文本数据打标签或添加相关信息。大量研究显示，利用大语言模型（LLMs）可替代人工标注，但现有工作多集中于经典NLP任务，而对需要专业知识的领域中，大模型作为标注者的表现仍缺乏深入探索。本文在三个高度专业化的领域中系统考察了大模型的标注能力，并从成本效益角度提出实用建议。据我们所知，这是首个针对大模型作为专家级标注者的系统性评估。
原文摘要 · Abstract (English)
Data annotation refers to the labeling or tagging of textual data with relevant information. A large body of works have reported positive results on leveraging LLMs as an alternative to human annotators. However, existing studies focus on classic NLP tasks, and the extent to which LLMs as data annotators perform in domains requiring expert knowledge remains underexplored. In this work, we investigate comprehensive approaches across three highly specialized domains and discuss practical suggestions from a cost-effectiveness perspective. To the best of our knowledge, we present the first systematic evaluation of LLMs as expert-level data annotators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。