利用标注者分歧提升文本分类效果,比只用多数标签更准确。
Leveraging Annotator Disagreement for Text Classification
- 通过概率多标签、集成系统和指令微调三种方式利用标注分歧
- 在仇恨言论检测中多标签方法最佳,在辱骂对话检测中指令微调最优
- 用户调研表明多标签模型输出更贴近文本真实语义
在文本分类任务中,通常仅使用多数标注作为训练标签,尽管数据集包含多个标注者。这会忽略标注者评估中的细微差别与多元视角。本文提出并比较了三种利用标注者分歧的策略:基于概率的多标签方法、集成系统和指令微调。所有方法均在仇恨言论和辱骂性对话检测任务上进行评估,这两项任务本身具有高度主观性。为验证引入标注分歧的有效性,我们开展在线调研,对比多标签模型与仅用多数标签训练的基线模型。结果表明,在仇恨言论检测中,多标签方法表现最佳;在辱骂对话检测中,指令微调效果最优。调研结果还显示,多标签模型的输出被认为更准确地反映了文本内容。
原文摘要 · Abstract (English)
It is common practice in text classification to only use one majority label for model training even if a dataset has been annotated by multiple annotators. Doing so can remove valuable nuances and diverse perspectives inherent in the annotators' assessments. This paper proposes and compares three different strategies to leverage annotator disagreement for text classification: a probability-based multi-label method, an ensemble system, and instruction tuning. All three approaches are evaluated on the tasks of hate speech and abusive conversation detection, which inherently entail a high degree of subjectivity. Moreover, to evaluate the effectiveness of embracing annotation disagreements for model training, we conduct an online survey that compares the performance of the multi-label model against a baseline model, which is trained with the majority label. The results show that in hate speech detection, the multi-label method outperforms the other two approaches, while in abusive conversation detection, instruction tuning achieves the best performance. The results of the survey also show that the outputs from the multi-label models are considered a better representation of the texts than the single-label model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。