arXiv:2609.09048cs.CLcs.AI2026-09

审计方式比人口统计差异更能影响大模型决策结果

The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits

  • 通过固定审计设计测试36组对比,发现排名方式改变导致的偏差远大于人口属性影响
  • 40,726条请求测试中,36组预期差异均未通过校正,多数效应仅为原文报告的一半
  • 模型对审计结构敏感,首列候选人获益等同于或超过种族偏差效应

语言模型是否表现出人口统计偏见,取决于审计提问方式。一项慈善援助基准显示,同一模型在逐个评分时偏好少数群体申请人,而在并列排名时则惩罚部分申请人。我们检验该反转现象是否适用于招聘、贷款和医疗分诊:共测试40,726条请求,使用仅姓名不同的申请材料,且主测试设计在数据收集前已固定。结果不支持该反转。36项预设对比中无一通过校正。评分优势的符号保持不变,但幅度约为原报告的一半;精度扩展分析表明,任何招聘排名惩罚均低于报告效应,而贷款与分诊排名下限高于此边际,因此在招聘排名及所有三领域评分中结论明确为非显著。植入的差异与注入程度一致,且在原始援助材料上实现方向性复现,验证了这些零结果。审计设计的影响远超人口统计因素:模型几乎总能识别透明审计,对相同内容比较无论差异是种族还是爱好都视为等价,且首列候选人获益程度与所测任何人口效应相当。审计结果反映的是审计构造,而非人口偏见。

原文摘要 · Abstract (English)

Whether a language model looks demographically biased can depend on how the audit asks its question. A charitable-aid benchmark reports that the same models favor minority applicants when rating requests one at a time and penalize some when ranking side by side. We test whether that reversal generalizes to hiring, lending, and medical triage: 40,726 requests to five models, applications differing only in the applicant's name, and a primary test fixed before collection. It does not. None of 36 planned contrasts survives correction. The rating advantage keeps its sign at roughly half the published size, and a precision extension bounds any hiring ranking penalty below the published effect, though the lending and triage ranking floors sit above that margin, so the exclusion is conclusive for hiring ranking and for rating in all three domains only. Planted disparities tracking their injected sizes and a directional replication on the original aid materials bound these nulls. The audit is livelier than the demographics: models recognize transparent audits nearly always, tie every identical-content comparison whether the varying detail is race or a hobby, and reward first-listed candidates as much as any demographic effect we measure. Audit verdicts reflect audit construction more than demographic bias.

大模型评估审计设计偏见检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。