用大模型生成的平衡句子提升预训练模型公平性
Bridging the Fairness Gap: Enhancing Pre-trained Models with LLM-Generated Sentences
- 用大模型生成语义丰富且属性均衡的句子增强模型
- 经因果分析筛选后，性别偏见显著降低，语言能力保持不变
- 适合关注模型公平性与可解释性的研究者
预训练语言模型（PLMs）在包含固有性别偏见的数据上训练，导致不良影响。传统去偏方法依赖外部语料，但其质量、多样性或人口分布可能失衡，影响去偏效果。随着大语言模型（LLM）知识广博，我们提出通过吸收连贯、属性平衡且语义丰富的句子来增强PLMs的公平性（Fair-Gender）。然而，这些句子因对齐问题和负迁移风险，不能直接用于去偏。为此，我们采用因果分析估计因果效应，过滤不一致句子，识别并保留对齐句子融入PLMs，确保正向迁移。实验表明，该方法显著降低PLMs中的性别偏见，同时保持语言表达能力。
原文摘要 · Abstract (English)
Pre-trained language models (PLMs) are trained on data that inherently contains gender biases, leading to undesirable impacts. Traditional debiasing methods often rely on external corpora, which may lack quality, diversity, or demographic balance, affecting the effectiveness of debiasing. With the rise of large language models and their extensive knowledge, we propose enhancing fairness (Fair-Gender) in PLMs by absorbing coherent, attribute-balanced, and semantically rich sentences. However, these sentences cannot be directly used for debiasing due to alignment issues and the risk of negative transfer. We address this by applying causal analysis to estimate causal effects, filtering out unaligned sentences, and identifying aligned ones for incorporation into PLMs, thereby ensuring positive transfer. Experiments show that our approach significantly reduces gender biases in PLMs while preserving their language expressiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。