揭示公平性评估在数据偏差下的脆弱性，提供可量化敏感性的新分析框架。
The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning
- 引入因果敏感性分析，评估公平性指标对数据偏差的响应
- 在14个数据集上验证，微小偏差即可使公平性结论失效
- 适合关注模型公平性可靠性的研究人员和实践者
公平性度量是公平机器学习（FairML）中的核心工具，用于判断模型在某种意义上是否‘公平’。然而，现实数据通常存在多种测量偏差和其他假设违背，可能导致公平性评估失去意义。本文将因果敏感性分析工具引入公平机器学习领域，提出一个通用框架：(1) 适用于任意组合的公平性度量与偏差设定；(2) 支持多类偏差组合，呈现非线性敏感性；(3) 可灵活编码领域特定约束与假设。基于该框架，我们在3种分类器、14个典型公平性数据集上分析了最常见的平等性度量的敏感性。结果揭示，公平性评估对微小数据偏差极度脆弱。研究证明，因果敏感性分析是评估平等性度量信息量不可或缺的工具。代码仓库已公开：https://github.com/Jakefawkes/fragile_fair。
原文摘要 · Abstract (English)
Fairness metrics are a core tool in the fair machine learning literature (FairML), used to determine that ML models are, in some sense, "fair". Real-world data, however, are typically plagued by various measurement biases and other violated assumptions, which can render fairness assessments meaningless. We adapt tools from causal sensitivity analysis to the FairML context, providing a general framework which (1) accommodates effectively any combination of fairness metric and bias that can be posed in the "oblivious setting"; (2) allows researchers to investigate combinations of biases, resulting in non-linear sensitivity; and (3) enables flexible encoding of domain-specific constraints and assumptions. Employing this framework, we analyze the sensitivity of the most common parity metrics under 3 varieties of classifier across 14 canonical fairness datasets. Our analysis reveals the striking fragility of fairness assessments to even minor dataset biases. We show that causal sensitivity analysis provides a powerful and necessary toolkit for gauging the informativeness of parity metric evaluations. Our repository is available here: https://github.com/Jakefawkes/fragile_fair.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。