构建首个德语性别公平文本数据集，揭示语言变化对模型预测的显著影响。
The Lou Dataset -- Exploring the Impact of Gender-Fair Language in German Text Classification
- 构建7个任务的德语性别公平改写数据集Lou
- 模型预测出现标签翻转、置信度下降、注意力模式改变
- 适用于关注语言公平性与模型鲁棒性的研究者
性别公平语言是德语中一种不断演进的表达方式，通过使用包容性或中性形式促进性别平等。然而，目前缺乏评估该语言变化对语言模型（LMs）分类性能影响的资源，因为这些模型可能未在相关变体上训练。为此，我们提出了Lou——首个覆盖七个任务（如立场检测、毒性分类）的高质量德语文本改写数据集。在Lou上评估16个单语和多语语言模型发现，性别公平语言显著改变了模型预测：导致标签翻转、置信度降低、注意力模式改变。尽管如此，原始与改写样本的模型排名无显著差异，表明现有评估仍具有效性。本研究为德语文本分类提供了初步洞察，且多语和英文模型中观察到一致模式，提示结论可能适用于其他语言。
原文摘要 · Abstract (English)
Gender-fair language, an evolving German linguistic variation, fosters inclusion by addressing all genders or using neutral forms. Nevertheless, there is a significant lack of resources to assess the impact of this linguistic shift on classification using language models (LMs), which are probably not trained on such variations. To address this gap, we present Lou, the first dataset featuring high-quality reformulations for German text classification covering seven tasks, like stance detection and toxicity classification. Evaluating 16 mono- and multi-lingual LMs on Lou shows that gender-fair language substantially impacts predictions by flipping labels, reducing certainty, and altering attention patterns. However, existing evaluations remain valid, as LM rankings of original and reformulated instances do not significantly differ. While we offer initial insights on the effect on German text classification, the findings likely apply to other languages, as consistent patterns were observed in multi-lingual and English LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。