构建多语言印度语境下的问答偏见评测基准，揭示本土化偏见问题。
BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Context
- 基于印度多元文化设计跨语言偏见评测集，覆盖8种语言与13类社会群体。
- 数据量达39万+样本，发现印地语等本土语言偏见更显著，超越英语模型。
- 适用于评估多语言大模型在印度社会中的公平性，推动本地化AI伦理研究。
评估语言模型中的社会偏见对确保AI系统公平性至关重要，可避免强化有害刻板印象。现有基准如偏见问答评测（BBQ）主要聚焦西方语境，难以适配印度背景。为此，我们提出BharatBBQ，一个针对印地语、英语、马拉地语、孟加拉语、泰米尔语、泰卢固语、奥里亚语和阿萨姆语的文化适配评测集，涵盖13个社会类别（含3个交叉群体），反映印度社会文化中的普遍偏见。原始数据集包含49,108个单语言样本，通过翻译与人工验证扩展至392,864个跨语言样本。我们在零样本与少样本设置下评估五类多语言语言模型家族的偏见与刻板印象得分。结果表明，各类语言与社会类别中均存在持续偏见，且在印度语言中常比英语中更严重，凸显了建立语言与文化双重适配的评测基准的重要性。
原文摘要 · Abstract (English)
Evaluating social biases in language models (LMs) is crucial for ensuring fairness and minimizing the reinforcement of harmful stereotypes in AI systems. Existing benchmarks, such as the Bias Benchmark for Question Answering (BBQ), primarily focus on Western contexts, limiting their applicability to the Indian context. To address this gap, we introduce BharatBBQ, a culturally adapted benchmark designed to assess biases in Hindi, English, Marathi, Bengali, Tamil, Telugu, Odia, and Assamese. BharatBBQ covers 13 social categories, including 3 intersectional groups, reflecting prevalent biases in the Indian sociocultural landscape. Our dataset contains 49,108 examples in one language that are expanded using translation and verification to 392,864 examples in eight different languages. We evaluate five multilingual LM families across zero and few-shot settings, analyzing their bias and stereotypical bias scores. Our findings highlight persistent biases across languages and social categories and often amplified biases in Indian languages compared to English, demonstrating the necessity of linguistically and culturally grounded benchmarks for bias evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。