基于拉美学校真实教学场景，构建了4.6万条社会偏见语料库。
HESEIA: A community-based dataset for evaluating social biases in large language models, co-designed in real school settings in Latin America
- 由370名教师与5370名学生共同设计，融合教育经验与本地情境
- 涵盖189所拉丁美洲学校，覆盖多个人口学维度与学科领域
- 揭示当前大模型未识别的隐性偏见，助力教育公平评估
现有大语言模型社会偏见评估资源多脱离受偏见影响社区，缺乏参与式设计。本文提出HESEIA，一个包含46,499条句子的数据集，源自一场面向370名高中生教师和5,370名学生的专业发展课程，覆盖189所拉丁美洲学校。与现有基准不同，HESEIA捕捉跨多个社会人口维度的交叉性偏见，并反映教育者的真实经历与教学专长。教师通过最小对（minimal pairs）构造与学科及社区相关的刻板印象语句。数据集在人口学特征和知识领域上均展现多样性。实验表明，该数据集包含比以往数据集更多当前大模型未能识别的偏见。HESEIA可支持基于教育社区的偏见评估。
原文摘要 · Abstract (English)
Most resources for evaluating social biases in Large Language Models are developed without co-design from the communities affected by these biases, and rarely involve participatory approaches. We introduce HESEIA, a dataset of 46,499 sentences created in a professional development course. The course involved 370 high-school teachers and 5,370 students from 189 Latin-American schools. Unlike existing benchmarks, HESEIA captures intersectional biases across multiple demographic axes and school subjects. It reflects local contexts through the lived experience and pedagogical expertise of educators. Teachers used minimal pairs to create sentences that express stereotypes relevant to their school subjects and communities. We show the dataset diversity in term of demographic axes represented and also in terms of the knowledge areas included. We demonstrate that the dataset contains more stereotypes unrecognized by current LLMs than previous datasets. HESEIA is available to support bias assessments grounded in educational communities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。