用大模型替代人工标注，成本低但一致性中等。
LLMs as Span Annotators: A Comparative Study of LLMs and Humans
- 让大模型直接做文本片段标注，对比人类专家表现
- 模型标注错误率与专业众包人员相当，仅中等一致度
- 标注成本仅为人工的几分之一，适合大规模应用
片段标注（span annotation）可对文本特定部分进行细粒度评估，适用于单一评分指标无法提供有效反馈的场景。以往主要依赖人工标注或微调模型完成。本文研究大语言模型（LLMs）能否替代人工标注者，在三项任务上比较了LLMs与熟练人类标注者的性能：数据到文本生成评估、翻译错误识别和宣传手法检测。结果表明，总体上LLMs与人类标注者间仅达到中等水平的标注一致性（IAA）。然而，模型犯错频率与熟练众包工作者相当。同时，LLMs的标注成本仅为人类的几分之一。研究发布了超过4万条模型与人类的标注数据集，供后续研究使用。
原文摘要 · Abstract (English)
Span annotation - annotating specific text features at the span level - can be used to evaluate texts where single-score metrics fail to provide actionable feedback. Until recently, span annotation was done by human annotators or fine-tuned models. In this paper, we study whether large language models (LLMs) can serve as an alternative to human annotators. We compare the abilities of LLMs to skilled human annotators on three span annotation tasks: evaluating data-to-text generation, identifying translation errors, and detecting propaganda techniques. We show that overall, LLMs have only moderate inter-annotator agreement (IAA) with human annotators. However, we demonstrate that LLMs make errors at a similar rate as skilled crowdworkers. LLMs also produce annotations at a fraction of the cost per output annotation. We release the dataset of over 40k model and human span annotations for further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。