静态词嵌入的内在偏见指标能预测部分场景下的系统偏见,但非普适。
Analyzing Correlations Between Intrinsic and Extrinsic Bias Metrics of Static Word Embeddings With Their Measuring Biases Aligned
- 通过匹配外在偏见数据集中的关键词,统一衡量标准
- 部分外在偏见任务中内在指标与结果呈中高相关
- 适用于特定场景偏见分析,不适用于所有模型和任务
我们评估静态词嵌入的内在偏见度量是否能预测自然语言处理(NLP)系统是否表现出偏见行为。词嵌入将词汇意义表示为实数向量,但也会习得社会偏见如刻板印象。内在偏见度量通过向量特征判断偏见,而外在偏见度量则检验使用该嵌入训练的NLP系统是否偏倚。先前研究发现常见内在度量与外在度量无相关性。然而,两者通常衡量不同类型的偏见,这让我们质疑相关性缺失是否真实。本文从外在偏见数据集中提取特征词,分析其与内在偏见度量的相关性,确保二者衡量相同偏见。结果显示,某些外在偏见度量存在中到高度相关,其他则无显著相关。这表明内在偏见度量仅在特定场景下可预测系统偏见。实验代码已开源于GitHub。
原文摘要 · Abstract (English)
We examine the abilities of intrinsic bias metrics of static word embeddings to predict whether Natural Language Processing (NLP) systems exhibit biased behavior. A word embedding is one of the fundamental NLP technologies that represents the meanings of words through real vectors, and problematically, it also learns social biases such as stereotypes. An intrinsic bias metric measures bias by examining a characteristic of vectors, while an extrinsic bias metric checks whether an NLP system trained with a word embedding is biased. A previous study found that a common intrinsic bias metric usually does not correlate with extrinsic bias metrics. However, the intrinsic and extrinsic bias metrics did not measure the same bias in most cases, which makes us question whether the lack of correlation is genuine. In this paper, we extract characteristic words from datasets of extrinsic bias metrics and analyze correlations with intrinsic bias metrics with those words to ensure both metrics measure the same bias. We observed moderate to high correlations with some extrinsic bias metrics but little to no correlations with the others. This result suggests that intrinsic bias metrics can predict biased behavior in particular settings but not in others. Experiment codes are available at GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。