用鲁棒优化方法在微调时降低大模型偏见，效果好且不伤性能。
RobustDebias: Debiasing Language Models using Distributionally Robust Optimization
- 引入分布鲁棒优化，微调阶段抑制偏见放大
- 多场景测试显示偏见显著降低，性能损失小
- 适用于各类数据集和任务，尤其适合常用BERT模型
预训练语言模型常表现出社会偏见与刻板印象。以往去偏方法多聚焦于预训练阶段修改嵌入空间，难以扩展至大模型。在特定任务数据上微调模型不仅会降低性能，还可能放大数据中的偏见。本文针对微调过程中的偏见放大问题，提出一种新机制 RobustDebias，将分布鲁棒优化（DRO）应用于 BERT 模型的微调阶段，以实现跨多个群体的去偏。该方法在掩码语言建模（MLM）微调中有效减少偏见，并可泛化至任意数据集或任务。大量实验表明，该方法在多种语言模型上均实现显著偏见缓解，同时保持极小性能损失。
原文摘要 · Abstract (English)
Pretrained language models have been shown to exhibit biases and social stereotypes. Prior work on debiasing these models has largely focused on modifying embedding spaces during pretraining, which is not scalable for large models. Fine-tuning pretrained models on task-specific datasets can both degrade model performance and amplify biases present in the fine-tuning data. We address bias amplification during fine-tuning rather than costly pretraining, focusing on BERT models due to their widespread use in language understanding tasks. While Empirical Risk Minimization effectively optimizes downstream performance, it often amplifies social biases during fine-tuning. To counter this, we propose \textit{RobustDebias}, a novel mechanism which adapts Distributionally Robust Optimization (DRO) to debias language models during fine-tuning. Our approach debiases models across multiple demographics during MLM fine-tuning and generalizes to any dataset or task. Extensive experiments on various language models show significant bias mitigation with minimal performance impact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。