用大模型辅助标注框架语义，提升多样性且不牺牲覆盖率。
Evaluating the Impact of LLM-Assisted Annotation in a Perspectivized Setting: the Case of FrameNet Annotation
- 混合半自动标注：人类+大模型协作，提升标注多样性
- 相比纯人工，半自动标注覆盖度相当但多样性更高
- 纯自动标注虽快但质量差，不适用于高质量语义标注
将基于大语言模型的工具用于加速或替代人工创建语言资源和数据集已成为现实。然而，尽管这类工具在语言学研究中具有潜力，针对其在视角化自然语言处理范式下对标注数据集创建性能与影响的系统性评估仍显不足。本文通过大规模实验评估了利用大模型进行类FrameNet语义角色标注的（半）自动化效果。实验比较了三种设置：纯人工、全自动和半自动标注。结果表明，相较于纯人工标注，半自动标注在帧多样性上显著提升，覆盖度保持相当；而全自动标注除标注时间外，各项指标均表现较差。
原文摘要 · Abstract (English)
The use of LLM-based applications as a means to accelerate and/or substitute human labor in the creation of language resources and dataset is a reality. Nonetheless, despite the potential of such tools for linguistic research, comprehensive evaluation of their performance and impact on the creation of annotated datasets, especially under a perspectivized approach to NLP, is still missing. This paper contributes to reduction of this gap by reporting on an extensive evaluation of the (semi-)automatization of FrameNet-like semantic annotation by the use of an LLM-based semantic role labeler. The methodology employed compares annotation time, coverage and diversity in three experimental settings: manual, automatic and semi-automatic annotation. Results show that the hybrid, semi-automatic annotation setting leads to increased frame diversity and similar annotation coverage, when compared to the human-only setting, while the automatic setting performs considerably worse in all metrics, except for annotation time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。