构建首个孟加拉语伪社会偏见评测集，揭示大模型在多元社会属性上的刻板印象。
BanStereoSet: A Dataset to Measure Stereotypical Social Biases in LLMs for Bangla
- 基于英文偏见数据本地化，生成涵盖9类社会偏见的孟加拉语语料
- 1,194条句子显示多语言大模型存在显著性别、种姓、宗教等偏见
- 为孟加拉语场景下公平语言技术开发提供关键评测工具
本研究提出 BanStereoSet，一个用于评估孟加拉语多语言大模型中刻板社会偏见的数据集。为拓展偏见研究从英语中心数据集的局限，我们对 StereoSet、IndiBias 及 Kamruzzaman 等人的数据集内容进行本地化处理，构建出适用于孟加拉语社区的资源。该数据集包含1,194条句子，覆盖种族、职业、性别、年龄歧视、外貌、职业外貌、地区、种姓和宗教共9类偏见。该数据集不仅可用于衡量多语言大模型的偏见水平，还支持跨社会维度的刻板偏见分析，有助于推动孟加拉国语境下的更公平语言技术发展。对多个语言模型的分析表明其存在显著偏见，凸显了构建文化与语言适配数据集的必要性。
原文摘要 · Abstract (English)
This study presents BanStereoSet, a dataset designed to evaluate stereotypical social biases in multilingual LLMs for the Bangla language. In an effort to extend the focus of bias research beyond English-centric datasets, we have localized the content from the StereoSet, IndiBias, and Kamruzzaman et. al.'s datasets, producing a resource tailored to capture biases prevalent within the Bangla-speaking community. Our BanStereoSet dataset consists of 1,194 sentences spanning 9 categories of bias: race, profession, gender, ageism, beauty, beauty in profession, region, caste, and religion. This dataset not only serves as a crucial tool for measuring bias in multilingual LLMs but also facilitates the exploration of stereotypical bias across different social categories, potentially guiding the development of more equitable language technologies in Bangladeshi contexts. Our analysis of several language models using this dataset indicates significant biases, reinforcing the necessity for culturally and linguistically adapted datasets to develop more equitable language technologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。