测试大模型在多种真实场景下的反穆斯林偏见，发现推理和代理行为会加剧偏见。
MIRAGE: Auditing Anti-Muslim Bias in Frontier LLMs Across Reasoning, Agentic, and Time-Coupled Conditions
- 设计三类真实部署场景评估偏见：直接生成、思维链推理、模拟代理决策。
- 思维链使穆斯林暴力关联上升12%至34%，代理决策对穆斯林案件更不利，差距达9%到22%。
- 偏见随新闻背景时间变化而增强，近期冲突内容使偏见上升18%至27%，适合安全与伦理研究者使用。
五年来，大语言模型中的持续反穆斯林偏见虽被发现，但多数评估仍局限于单轮提示生成，无法反映前沿模型的实际部署。我们提出 extbf{MIRAGE}（穆斯林身份推理与代理生成评估基准），包含1,200个提示，覆盖三种部署现实条件：直接生成、思维链推理，以及在内容审核、贷款筛选、难民申请摘要和招聘筛查中的模拟代理决策。在六款前沿模型中，我们发现：(i) 思维链推理使穆斯林-暴力关联比直接生成高12%至34%；(ii) 代理决策在相同证据下，穆斯林案例与匹配非穆斯林案例间存在9%至22个百分点的不对称性；(iii) 偏见显著受检索新闻内容的时间影响，在近期冲突背景下偏见上升18%至27%。现有基于提示的缓解方法在跨条件迁移中表现不佳，仅抑制直接生成偏见，难以消除代理不对称性。我们公开发布 MIRAGE 及开放评估工具，支持针对性缓解研究。
原文摘要 · Abstract (English)
Five years after the discovery of persistent anti-Muslim bias in large language models, most evaluations remain confined to single-turn prompt completion, a setting that no longer reflects how frontier LLMs are deployed. We introduce \textbf{MIRAGE} (Muslim-Identity Reasoning and Agentic Generation Evaluation), a benchmark of 1{,}200 prompts spanning three deployment-realistic conditions: direct completion, chain-of-thought reasoning, and simulated agentic decision-making across content moderation, lending triage, refugee claim summarization, and hiring screens. Across six frontier models, we find that (i) chain-of-thought reasoning \emph{amplifies} rather than suppresses Muslim-violence associations by 12--34\% relative to direct completion, (ii) agentic decisions exhibit a 9--22 percentage-point asymmetry between Muslim and matched non-Muslim cases on identical evidence, and (iii) bias is sharply time-coupled to retrieved news context, increasing 18--27\% under recent-conflict retrieval. Existing prompt-based mitigations transfer poorly across our three conditions, suppressing direct-completion bias while leaving agentic asymmetry largely intact. We release MIRAGE and an open evaluation harness to support targeted mitigation research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。