提出多层级性别包容性评估框架，量化大模型对非二元性别的支持程度。
Gender Inclusivity Fairness Index (GIFI): A Multilevel Framework for Evaluating Gender Diversity in Large Language Models
- 构建从指代到生成行为的多层评估体系，覆盖多种性别身份
- 在22个大模型中发现显著性别包容性差异，最大差距达47%
- 为生成模型性别公平性提供可复用的评测基准，适合伦理与安全研究者
我们提出了一个全面评估大语言模型（LLMs）性别公平性的框架，重点关注其对二元与非二元性别的处理能力。现有研究多聚焦于二元性别区分，本文引入全新的性别包容性公平指数（GIFI），量化大模型在性别多样性方面的表现。GIFI包含多层次评估：从简单探查模型对给定性别代词的响应，到测试不同性别假设下模型生成与认知行为的表现，揭示了与不同性别标识相关的偏见。我们在22个不同规模与能力的开源及专有大模型上进行了广泛评估，发现模型间性别包容性存在显著差异。本研究强调提升大模型包容性的必要性，为未来生成模型的性别公平性发展提供了关键基准。
原文摘要 · Abstract (English)
We present a comprehensive evaluation of gender fairness in large language models (LLMs), focusing on their ability to handle both binary and non-binary genders. While previous studies primarily focus on binary gender distinctions, we introduce the Gender Inclusivity Fairness Index (GIFI), a novel and comprehensive metric that quantifies the diverse gender inclusivity of LLMs. GIFI consists of a wide range of evaluations at different levels, from simply probing the model with respect to provided gender pronouns to testing various aspects of model generation and cognitive behaviors under different gender assumptions, revealing biases associated with varying gender identifiers. We conduct extensive evaluations with GIFI on 22 prominent open-source and proprietary LLMs of varying sizes and capabilities, discovering significant variations in LLMs' gender inclusivity. Our study highlights the importance of improving LLMs' inclusivity, providing a critical benchmark for future advancements in gender fairness in generative models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。