arXiv:2604.02951cs.CLcs.AI2026-04中稿 · AIED 2026被引 1

标注过程本身会提升标注者能力,尤其对专家更明显。

How Annotation Trains Annotators: Competence Development in Social Influence Recognition

论文配图:How Annotation Trains Annotators: Competence Development in Social Influence Recognition
图 1 · 摘自论文原文
  • 通过前后两次标注对比,分析标注者能力变化
  • 专家组标注质量提升显著,自评信心也增强
  • 这种能力提升会影响后续大模型的训练效果

人类数据标注常被视为客观参考,但涉及主观判断的任务中,标注者的能力可能随时间演变。本研究考察了社会影响识别任务中25名标注者(含专家与非专家)的胜任力发展情况。他们对1021段对话标注了20种社会影响策略,以及意图、反应和后果。初始150条文本在主标注前后各标注一次,用于对比。结合定量分析、定性评估、半结构化访谈与自评问卷,并用大模型在对比数据集上训练与评测,结果表明标注者自我感知胜任力和信心显著提升;标注质量变化显示,标注过程确实增强了标注者能力,且该效应在专家组中更为明显。此外,标注者能力的变化对基于其标注数据训练的大模型性能产生了可测量的影响。

原文摘要 · Abstract (English)

Human data annotation, especially when involving experts, is often treated as an objective reference. However, many annotation tasks are inherently subjective, and annotators' judgments may evolve over time. This study investigates changes in the quality of annotators' work from a competence perspective during a process of social influence recognition. The study involved 25 annotators from five different groups, including both experts and non-experts, who annotated a dataset of 1,021 dialogues with 20 social influence techniques, along with intentions, reactions, and consequences. An initial subset of 150 texts was annotated twice - before and after the main annotation process - to enable comparison. To measure competence shifts, we combined qualitative and quantitative analyses of the annotated data, semi-structured interviews with annotators, self-assessment surveys, and Large Language Model training and evaluation on the comparison dataset. The results indicate a significant increase in annotators' self-perceived competence and confidence. Moreover, observed changes in data quality suggest that the annotation process may enhance annotator competence and that this effect is more pronounced in expert groups. The observed shifts in annotator competence have a visible impact on the performance of LLMs trained on their annotated data.

数据标注认知演化专家能力大模型训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。