arXiv:2506.23033cs.LGstat.ML2025-06

当人口数据缺失时,公平性审计应披露哪些信息?

What Must a Fairness Audit Report When Demographic Data Is Incomplete?

  • 用基线对比揭示审计结果的可靠性
  • 数据缺失时,多数方法反而降低整体公平性
  • 需报告候选方案集与交叉群体影响

公平性审计是负责任机器学习部署的关键环节。然而,当依赖的受保护标签不完整时,审计应披露什么仍不明确。本文聚焦于审计发布的各类比率及其解读所需背景信息。我们为每项公布比率匹配了两个基线:一个隐藏受保护标签,另一个仅改变运行种子。在ACS/Folktables任务中,保留部分受保护标签的缺失设置导致选定缓解措施的变化小于一次普通重跑。在完全无受保护标签访问的情况下,候选方案退化为经验风险最小化,此时看似异常实则源于候选集构成。等几率阈值优化最常使交叉子群表现下降,但若仅保留审计员可接受的配置,该率会回归基线。其准确率损失对全体人群和最差子群影响均等,机制实质为“向下拉平”。唯一持续存在的效应是哪个子群成为最差者发生了变化。总体而言,审计比率必须连同解释所需的基线、候选集来源及交叉效应一并发布,才能作为模型部署证据。

原文摘要 · Abstract (English)

Fairness audits are a key component of responsible machine-learning deployment. Yet what such an audit must disclose, when the protected labels it depends on are incomplete, remains unsettled. In this work, we focused on the rates a fairness audit publishes and on what an oversight reader needs beside them. We paired every published rate with a matched baseline drawn from the same audit, one hiding protected labels and one varying only the run seed. Across ACS/Folktables tasks, missingness settings that kept some protected labels moved the selected mitigation less than an ordinary rerun did. At zero protected-label access, candidates collapsed to empirical risk minimization, so the apparent exception there reflected the candidate set's composition. Equalized-odds threshold optimization most often regressed an intersectional subgroup, but that rate fell back to its baseline once we kept only the configurations an auditor would accept. Any accuracy it lost fell on the population as heavily as on the worst-off cell, so the mechanism is levelling down. The one effect that survived was a change in which cell is worst-off. Overall, our results highlight that a published audit rate should be reported with the baseline needed to interpret it, the candidate set it came from, and its intersectional effects, before it is treated as evidence about a deployed model.

公平性审计数据缺失基准对比交叉群体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。