测试大模型代理在招聘、信贷等场景中是否因种族性别产生行动歧视。
AgentFairBench: Do LLM Agents Discriminate When They Act?
- 用反事实姓名变量设计多领域公平性测试，模拟种族与性别信号差异。
- 发现Claude Haiku 4.5在120组对比中无显著歧视，工具调用差异也未超采样噪声。
- 提供可复现的低成本评估工具，适合研究者与开发者验证模型公平性。
大型语言模型代理越来越多地参与实际决策（如筛选求职者、推荐信贷、患者分诊），但现有公平性评估仍停留在答案评分层面。本文提出AgentFairBench，一个低成本、可复现、跨多领域的模型代理行为公平性基准。基于偏见传导框架（BCF），覆盖招聘、信贷和医疗分诊三个受监管领域。使用合成的、人口统计中性的人物资料，在仅改变姓名所编码的种族×性别信号（沿Bertrand-Mullainathan范式）的反事实匹配组中，测试四种不同代理架构（直接输出、思维链、多代理协商、工具增强）下的表现。采用纯NumPy实现的评估框架，计算反事实翻转率、平均绝对得分差（MASD）、行动率差异和工具调用差异，并通过自助法置信区间、配对检验与错误发现率控制进行统计分析，每模型成本低于单美元。设有公开排行榜及私有保留集与污染检测器，支持外部模型提交。初步实验（864次决策+测试重测复制）揭示方法论教训：将六组得分分布与双运行噪声差异比较，会高估不公平性约2.4倍。在匹配的零假设基线与全组检验下，Claude Haiku 4.5无任何一对或多组对比显著超出采样噪声（120组两两对比和9组全组检验均未通过校正）；植入偏见测试确认该工具能有效检测真实偏差。贡献包括一个稳健、敏感、可落地的评估工具，匹配运算量的零假设方法，以及可扩展的开源资源。代码、数据与评估工具以开源许可发布，附匿名评审材料。
原文摘要 · Abstract (English)
Large language model (LLM) agents increasingly take actions (screening applicants, recommending credit, triaging patients), yet fairness for LLMs is still measured by grading answers. We introduce AgentFairBench, a cheap, reproducible, multi-domain benchmark for demographic disparity in the actions of LLM agents. Grounded in a companion framework, the Bias Conduction Framework (BCF, restated here), it spans three regulator-anchored domains: hiring, lending, and medical triage. Synthetic, demographic-neutral profiles are evaluated in counterfactual matched sets that vary only a name-coded race x gender signal (in the Bertrand Mullainathan tradition), under four agent scaffolds of increasing agency (direct, chain-of-thought, multi-agent deliberation, tool-augmented). A NumPy-only harness computes counterfactual flip rate, mean absolute score difference (MASD), action-rate disparity, and tool-invocation disparity, with bootstrap confidence intervals, paired tests, and false-discovery-rate control, for single-digit dollars per model. A live leaderboard with a held-out private split and a contamination canary admits external models by submission. Our pilot (864 decisions plus a test-retest replication) carries a methodological lesson: comparing a six-group score spread against a two-run noise difference overstates disparity by ~ 2.4X through statistic arity alone. Against an arity matched noise floor and an omnibus group test, claude haiku 4 5 shows no demographic effect above sampling noise (0 of 120 pairwise and 0 of 9 omnibus contrasts survive correction); a planted-bias test confirms the instrument detects disparity when present. The contribution is a sound, sensitive, adoption-ready instrument, the arity matched null methodology, and open artifacts to scale it. Code, data, and harness are released under open licenses, with an anonymized review artifact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。