LLM在零样本立场检测中受刻板印象影响，误判文本立场。
Are Stereotypes Leading LLMs' Zero-Shot Stance Detection ?
- 通过标注文本方言与可读性，分析模型决策偏差。
- 模型将大麻支持观点错误关联低复杂度文本，将非裔美语关联反特朗普立场。
- 揭示了大模型在敏感任务中的隐性偏见，适合关注AI伦理的研究者。
大型语言模型从预训练数据中继承刻板印象，导致在诸如仇恨言论检测或情感分析等自然语言处理任务中对特定社会群体产生偏差行为。令人惊讶的是，社区对立场检测方法中此类偏见的评估长期被忽视。立场检测涉及将陈述标记为针对特定目标的反对、支持或中立，是高度敏感的NLP任务，常与政治倾向相关。本文聚焦于大语言模型在零样本设置下进行立场检测时的偏见问题。我们自动标注现有立场检测数据集中的帖子，加入两个属性：特定群体的语言方言或口语特征，以及文本复杂度/可读性，以探究这些属性是否影响模型的立场判断。结果表明，大语言模型在立场检测任务中表现出显著的刻板印象，例如将支持大麻的观点错误关联至低复杂度文本，并将非裔美国人方言与反对唐纳德·特朗普的立场相联系。
原文摘要 · Abstract (English)
Large Language Models inherit stereotypes from their pretraining data, leading to biased behavior toward certain social groups in many Natural Language Processing tasks, such as hateful speech detection or sentiment analysis. Surprisingly, the evaluation of this kind of bias in stance detection methods has been largely overlooked by the community. Stance Detection involves labeling a statement as being against, in favor, or neutral towards a specific target and is among the most sensitive NLP tasks, as it often relates to political leanings. In this paper, we focus on the bias of Large Language Models when performing stance detection in a zero-shot setting. We automatically annotate posts in pre-existing stance detection datasets with two attributes: dialect or vernacular of a specific group and text complexity/readability, to investigate whether these attributes influence the model's stance detection decisions. Our results show that LLMs exhibit significant stereotypes in stance detection tasks, such as incorrectly associating pro-marijuana views with low text complexity and African American dialect with opposition to Donald Trump.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。