构建首个用于检测长文本中性别化与误性别化现象的基准数据集
ProText: A benchmark dataset for measuring (mis)gendering in long-form texts
- 设计三维度标注体系：主题名词、主题类别与代词类别
- 发现大模型在无明确性别线索时易默认异性恋规范，存在系统性偏见
- 适用于评估文本生成与摘要中的性别偏差，超越传统代词消解任务
我们提出ProText，一个用于衡量英语长文本中性别化与误性别化现象的基准数据集。该数据集涵盖三个维度：主题名词（姓名、职业、头衔、亲属称谓）、主题类别（刻板男性、刻板女性、中性/非性别化）和代词类别（阳性、阴性、中性、无）。数据集旨在评估大语言模型在摘要、重写等文本转换任务中的性别表现，突破传统代词消解基准和性别二元框架。通过小规模案例研究验证，仅使用两个提示和两个模型，即可揭示性别偏见、刻板印象、误性别化及性别化行为的细微模式。结果显示，在输入缺乏显式性别线索或模型默认异性恋假设时，系统性性别偏见尤为明显。
原文摘要 · Abstract (English)
We introduce ProText, a dataset for measuring gendering and misgendering in stylistically diverse long-form English texts. ProText spans three dimensions: Theme nouns (names, occupations, titles, kinship terms), Theme category (stereotypically male, stereotypically female, gender-neutral/non-gendered), and Pronoun category (masculine, feminine, gender-neutral, none). The dataset is designed to probe (mis)gendering in text transformations such as summarization and rewrites using state-of-the-art Large Language Models, extending beyond traditional pronoun resolution benchmarks and beyond the gender binary. We validated ProText through a mini case study, showing that even with just two prompts and two models, we can draw nuanced insights regarding gender bias, stereotyping, misgendering, and gendering. We reveal systematic gender bias, particularly when inputs contain no explicit gender cues or when models default to heteronormative assumptions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。