arXiv:2410.12217cs.CL2024-10EMNLP被引 6

通过融合标注者历史与问卷信息,提升文本毒性预测精度。

Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree

  • 基于嵌入的架构融合标注者历史、人口统计与问卷数据
  • 相比传统方法,预测准确率显著提升,尤其在标注不一致时
  • 问卷信息可替代真实人口统计,适合主观任务建模

当标注者意见不一致时,直接预测单个标注者的评分可捕捉传统标签聚合忽略的细微差异。本文提出三种方法:神经协同过滤(NCF)、上下文学习(ICL)及基于中间嵌入的架构,结合标注者特定信息进行毒性评分预测。研究发现NCF效果有限,但整合标注者历史、人口统计与问卷信息后,嵌入式架构和ICL均显著提升预测准确率,其中嵌入式架构表现最优。此外,若通过问卷信息推断人口统计特征,使用这些推断数据的效果与真实数据相当,表明人口统计信息提供的额外价值有限,其信息已基本包含在问卷响应中。该结果对主观自然语言处理任务中标注者建模的策略选择具有启示意义。

原文摘要 · Abstract (English)

When annotators disagree, predicting the labels given by individual annotators can capture nuances overlooked by traditional label aggregation. We introduce three approaches to predicting individual annotator ratings on the toxicity of text by incorporating individual annotator-specific information: a neural collaborative filtering (NCF) approach, an in-context learning (ICL) approach, and an intermediate embedding-based architecture. We also study the utility of demographic information for rating prediction. NCF showed limited utility; however, integrating annotator history, demographics, and survey information permits both the embedding-based architecture and ICL to substantially improve prediction accuracy, with the embedding-based architecture outperforming the other methods. We also find that, if demographics are predicted from survey information, using these imputed demographics as features performs comparably to using true demographic data. This suggests that demographics may not provide substantial information for modeling ratings beyond what is captured in survey responses. Our findings raise considerations about the relative utility of different types of annotator information and provide new approaches for modeling annotators in subjective NLP tasks.

毒性检测标注一致性嵌入模型主观任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。