用检测偏差所需样本量衡量算法偏见，样本越大越不偏。
Measures of classification bias derived from sample size analysis
- 通过统计检测偏差所需的样本量反推偏见程度
- 在两组人群中，误差率差异越难检测，偏见越小
- 相比误差差值和比值，该方法能更合理排序算法偏见
我们提出一种直观的算法分类偏见度量方法：若要统计上确认分类器在不同人口群体间存在误差率差异，所需样本量越大，则偏见越小。在两组人群的简单设定下，基于非参数误差率估计值e1和e2，使用卡方检验的近似样本量公式，验证了该度量的一些理想性质。我们将其与常用指标误差差值e2-e1和比值e2/e1进行比较，发现该方法可对算法偏见给出不同的排序，且具有优势。最后讨论表明，这些优良性质源于方法的本质特征，而非依赖于近似公式，因此在多于两个群体的复杂场景中也预期成立。
原文摘要 · Abstract (English)
We propose the use of a simple intuitive principle for measuring algorithmic classification bias: the significance of the differences in a classifier's error rates across the various demographics is inversely commensurate with the sample size required to statistically detect them. That is, if large sample sizes are required to statistically establish biased behavior, the algorithm is less biased, and vice versa. In a simple setting, we assume two distinct demographics, and non-parametric estimates of the error rates on them, e1 and e2, respectively. We use a well-known approximate formula for the sample size of the chi-squared test, and verify some basic desirable properties of the proposed measure. Next, we compare the proposed measure with two other commonly used statistics, the difference e2-e1 and the ratio e2/e1 of the error rates. We establish that the proposed measure is essentially different in that it can rank algorithms for bias differently, and we discuss some of its advantages over the other two measures. Finally, we briefly discuss how some of the desirable properties of the proposed measure emanate from fundamental characteristics of the method, rather than the approximate sample size formula we used, and thus, are expected to hold in more complex settings with more than two demographics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。