评测大模型在政治语境中的性别与立场偏见，发现其对女性议员识别错误率高。
Benchmarking Gender and Political Bias in Large Language Models
- 构建欧盟议会辩论与投票数据关联的基准数据集EuroParlVote
- 模型误判女性议员性别，且对女性发言者预测准确率下降
- 主流模型更倾向中立立场，对极左极右群体表现较差
我们提出EuroParlVote，一个用于评估大语言模型（LLMs）在政治敏感语境中表现的新基准。该数据集将欧洲议会辩论演讲与投票结果关联，并包含每位议员（MEP）的丰富人口统计信息，如性别、年龄、国籍和政治派别。基于此，我们评估了先进LLMs在性别分类与投票预测两个任务上的表现，揭示出系统性偏见。结果显示，模型常将女性议员误判为男性，且在模拟女性发言者投票时准确率降低。政治层面，模型倾向于支持中间派，对极左与极右派别表现不佳。商用模型如GPT-4o在鲁棒性与公平性上优于开源模型。我们已公开发布EuroParlVote数据集、代码与演示，以推动政治语境下NLP公平性与问责研究。
原文摘要 · Abstract (English)
We introduce EuroParlVote, a novel benchmark for evaluating large language models (LLMs) in politically sensitive contexts. It links European Parliament debate speeches to roll-call vote outcomes and includes rich demographic metadata for each Member of the European Parliament (MEP), such as gender, age, country, and political group. Using EuroParlVote, we evaluate state-of-the-art LLMs on two tasks -- gender classification and vote prediction -- revealing consistent patterns of bias. We find that LLMs frequently misclassify female MEPs as male and demonstrate reduced accuracy when simulating votes for female speakers. Politically, LLMs tend to favor centrist groups while underperforming on both far-left and far-right ones. Proprietary models like GPT-4o outperform open-weight alternatives in terms of both robustness and fairness. We release the EuroParlVote dataset, code, and demo to support future research on fairness and accountability in NLP within political contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。