用多元观点标注提升立场检测模型的可靠性
Embracing Diversity: A Multi-Perspective Approach with Soft Labels
- 引入多视角标注框架，融合不同背景的判断
- F1分数更高，但模型置信度更低，反映任务主观性
- 适合关注公平性和可解释性的自然语言处理研究者
先前研究表明，利用不同背景和人生经历带来的标注多样性，并将其融入模型学习，即多视角方法，有助于开发更负责任的模型。本文提出一种新框架，用于设计和评估立场检测任务中的视角感知模型，其中多位标注者针对争议话题给出立场判断。同时，我们公开了一个新数据集，包含人类与大模型的双重标注。实验结果表明，多视角方法在分类性能上优于传统单标签基准，获得更高的F1分数，但模型置信度更低，可能源于立场检测任务本身的高主观性。
原文摘要 · Abstract (English)
Prior studies show that adopting the annotation diversity shaped by different backgrounds and life experiences and incorporating them into the model learning, i.e. multi-perspective approach, contribute to the development of more responsible models. Thus, in this paper we propose a new framework for designing and further evaluating perspective-aware models on stance detection task,in which multiple annotators assign stances based on a controversial topic. We also share a new dataset established through obtaining both human and LLM annotations. Results show that the multi-perspective approach yields better classification performance (higher F1-scores), outperforming the traditional approaches that use a single ground-truth, while displaying lower model confidence scores, probably due to the high level of subjectivity of the stance detection task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。