评估荷兰语仇恨言论检测模型的公平性,发现加入对抗样本后性能与公平性双提升。
Towards Fairness Assessment of Dutch Hate Speech Detection
- 用大模型生成荷兰语对抗性数据,改进仇恨言论检测的公平性评估
- 模型在对抗数据上检测准确率提升,群体公平性指标改善显著
- 为非英语仇恨言论检测提供可复用的公平性评估框架,适合伦理与AI安全研究者
现有仇恨言论检测研究多集中于英语,且侧重模型开发。本研究首次评估荷兰语仇恨言论检测模型的反事实公平性,重点关注基于Transformer的模型。首先,我们构建了反映社会语境的荷兰语社会群体术语列表;其次,利用大模型和人工组替换(MGS)、句子对数似然(SLL)策略生成荷兰语对抗性数据,但发现其在语法和语境连贯性方面存在挑战;第三,以对抗数据微调基线模型并评估其检测性能;第四,采用反事实标记公平性(CTF)、均等机会和人口均等性等指标评估模型公平性。结果表明,模型在仇恨言论检测、平均反事实公平性和群体公平性方面均有提升。该工作填补了荷兰语仇恨言论检测中反事实公平性的研究空白,为提升模型性能与公平性提供了实用建议。
原文摘要 · Abstract (English)
Numerous studies have proposed computational methods to detect hate speech online, yet most focus on the English language and emphasize model development. In this study, we evaluate the counterfactual fairness of hate speech detection models in the Dutch language, specifically examining the performance and fairness of transformer-based models. We make the following key contributions. First, we curate a list of Dutch Social Group Terms that reflect social context. Second, we generate counterfactual data for Dutch hate speech using LLMs and established strategies like Manual Group Substitution (MGS) and Sentence Log-Likelihood (SLL). Through qualitative evaluation, we highlight the challenges of generating realistic counterfactuals, particularly with Dutch grammar and contextual coherence. Third, we fine-tune baseline transformer-based models with counterfactual data and evaluate their performance in detecting hate speech. Fourth, we assess the fairness of these models using Counterfactual Token Fairness (CTF) and group fairness metrics, including equality of odds and demographic parity. Our analysis shows that models perform better in terms of hate speech detection, average counterfactual fairness and group fairness. This work addresses a significant gap in the literature on counterfactual fairness for hate speech detection in Dutch and provides practical insights and recommendations for improving both model performance and fairness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。