检测大模型生成人物形象时的偏见，发现越大的模型未必越公平。
When LLMs Imagine People: A Human-Centered Persona Brainstorm Audit for Bias and Fairness in Creative Applications
- 提出可扩展的Persona Brainstorm Audit方法，评估多维度身份偏见。
- 12个模型生成12万个人物，发现偏见随版本非线性变化，新模型未必更公平。
- 能揭示单一指标忽略的交叉身份偏见，适合评估创意类AI公平性。
用于创意工作流的大语言模型可能强化刻板印象并延续不平等，因此公平性审计至关重要。现有方法依赖受限任务和固定基准，未覆盖开放式的创造性输出。本文提出Persona Brainstorm Audit（PBA），一种可扩展且易于拓展的审计方法，用于在开放式人物生成中检测多重交叉身份与社会角色的偏见。PBA采用考虑自由度的归一化Cramér's V量化偏见，生成可解释的严重程度标签，实现模型与维度间的公平比较。对12个LLM（共120,000个角色，16个偏见维度）的应用表明，偏见随模型代际非线性演变：更大更新的模型并非始终更公平，某些早期减弱的偏见在后续版本中重现。交叉分析揭示了单轴指标掩盖的差异，即个别维度看似公平，组合后却表现出高偏见。鲁棒性分析显示，PBA在不同样本量、角色扮演提示和去偏提示下仍保持稳定，验证其在大模型公平性审计中的可靠性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) used in creative workflows can reinforce stereotypes and perpetuate inequities, making fairness auditing essential. Existing methods rely on constrained tasks and fixed benchmarks, leaving open-ended creative outputs unexamined. We introduce the Persona Brainstorm Audit (PBA), a scalable and easy to extend auditing method for bias detection across multiple intersecting identity and social roles in open-ended persona generation. PBA quantifies bias using degree-of-freedom-aware normalized Cramér's V, producing interpretable severity labels that enable fair comparison across models and dimensions. Applying PBA to 12 LLMs (120,000 personas, 16 bias dimensions), we find that bias evolves nonlinearly across model generations: larger and newer models are not consistently fairer, and biases that initially decrease can resurface in later releases. Intersectional analysis reveals disparities hidden by single-axis metrics, where dimensions appearing fair individually can exhibit high bias in combination. Robustness analyses show PBA remains stable under varying sample sizes, role-playing prompts, and debiasing prompts, establishing its reliability for fairness auditing in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。