arXiv:2601.17146stat.MEcs.CY2026-01

提出统计检验方法,验证算法是否预测了意图之外的变量。

Falsifying Discriminant Validity of Predictive Algorithms

  • 基于反事实检验思想,比较算法对目标与禁止变量的预测误差。
  • 在招生场景中,算法能区分性别但无法区分种族。
  • 适合用于部署前评估算法有效性,尤其关注隐含偏见。

针对预测模型无意中预测非预期结果的问题,本文提出一种可验证的判别效度检验框架。该框架通过对比校准后的预测损失,判断算法是否在预期结果上表现优于禁止性代理变量。借鉴因果推断、计量经济学和心理测量学中的反事实方法,框架采用非参数假设检验,对数据生成过程假设极少。在招生场景中,结果显示算法具备性别判别效度,但不具备种族判别效度;在刑事司法场景中,框架揭示其局限性,并强调需结合其他方法评估构念效度与外部效度。该方法可作为部署前的有效性早期检查工具。

原文摘要 · Abstract (English)

Empirical investigations into unintended model behavior often show that the algorithm is predicting another outcome than what was intended. These exposés highlight the need to identify when algorithms predict unintended quantities - ideally before deploying them into consequential settings. We propose a falsification framework that provides a principled statistical test for discriminant validity: the requirement that an algorithm predict intended outcomes better than impermissible ones. Drawing on falsification practices from causal inference, econometrics, and psychometrics, our framework compares calibrated prediction losses across outcomes to assess whether the algorithm exhibits discriminant validity with respect to a specified impermissible proxy. In settings where the target outcome is difficult to observe, multiple permissible proxy outcomes may be available; our framework accommodates both this setting and the case with a single permissible proxy. Throughout we use nonparametric hypothesis testing methods that make minimal assumptions on the data-generating process. We illustrate the method in an admissions setting, where the framework establishes discriminant validity with respect to gender but fails to establish discriminant validity with respect to race. This demonstrates how falsification can serve as an early validity check. We also provide analysis in a criminal justice setting, where we highlight the limitations of our framework and emphasize the need for complementary approaches to assess other aspects of construct validity and external validity.

算法验证判别效度公平性检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。