用无监督工具在无身份数据下发现教育风险算法的歧视问题
Auditing a Dutch Public Sector Risk Profiling Algorithm Using an Unsupervised Bias Detection Tool
- 用无监督方法检测算法偏见，无需依赖敏感身份数据
- 发现非欧洲移民背景学生与荷兰/欧洲背景学生间存在显著风险评分差异
- 开源工具包供公众审计，适合政策制定者和伦理审查人员
算法正被广泛用于辅助或自动化决策，但研究表明其可能在受法律保护的群体中表现出偏见。然而，由于隐私法规限制，组织或外部审计方可能无法获取这些群体的数据。本文研究在缺乏人口统计信息的情况下，使用无监督偏见检测工具进行审计。我们与荷兰教育执行机构合作，对2012至2023年间全国范围内用于为大学生分配风险评分的算法进行了审计，覆盖超过25万学生。该无监督工具揭示了非欧洲移民背景学生与荷兰或欧洲移民背景学生之间的已知差距。本研究贡献在于：（1）在真实世界、大规模且高风险的政府决策流程中评估偏见；（2）将无监督偏见检测工具开源，供他人用于完成偏见审计。工作为人类专家开展算法决策中潜在歧视的审议提供了起点。
原文摘要 · Abstract (English)
Algorithms are increasingly used to automate or aid human decisions, yet recent research shows that these algorithms may exhibit bias across legally protected demographic groups. However, data on these groups may be unavailable to organizations or external auditors due to privacy legislation. This paper studies bias detection using an unsupervised bias detection tool when data on demographic groups are unavailable. We collaborated with the Dutch Executive Agency for Education to audit an algorithm that was used to assign risk scores to college students at the national level in the Netherlands between 2012-2023. Our audit covers more than 250,000 students across the country. The unsupervised bias detection tool highlights known disparities between students with a non-European migration background and students with a Dutch or European-migration background. Our contributions are two-fold: (1) we assess bias in a real-world, large-scale, and high-stakes decision-making process by a governmental organization; (2) we provide the unsupervised bias detection tool in an open-source library for others to use to complete bias audits. Our work serves as a starting point for a deliberative assessment by human experts to evaluate potential discrimination in algorithmic decision-making.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。