针对巴基斯坦文化设计的问答偏见测试集，验证多语言模型在本地语境下的公平性表现。
PakBBQ: A Culturally Adapted Bias Benchmark for QA
- 构建涵盖8类偏见的双语（英/乌尔都）问答数据集
- 发现明确消歧可提升12%准确率，负向提问减少刻板印象
- 揭示乌尔都语模型比英语模型更少偏见，适合低资源语言研究
随着大语言模型在各类应用中的广泛使用，确保其对所有用户群体的公平性至关重要。然而，多数模型基于以西方为中心的数据训练和评估，对低资源语言和区域背景关注不足。为此，我们提出PakBBQ，作为原始问答偏见基准（BBQ）数据集的文化与区域适配版本。PakBBQ包含超过214个模板、17180组问答对，覆盖年龄、残疾、外貌、性别、社会经济地位、宗教、地区归属和语言正式度共8个偏见维度，涵盖英语和乌尔都语。我们在模糊与明确消歧情境下，以及负面与非负面问题表述中评估多个多语言大模型。实验表明：(i) 明确消歧使平均准确率提升12%；(ii) 乌尔都语模型表现出更一致的反偏见行为；(iii) 负面提问框架显著降低刻板回答。这些发现凸显了上下文化基准与简单提示工程在低资源场景中缓解偏见的重要性。
原文摘要 · Abstract (English)
With the widespread adoption of Large Language Models (LLMs) across various applications, it is empirical to ensure their fairness across all user communities. However, most LLMs are trained and evaluated on Western centric data, with little attention paid to low-resource languages and regional contexts. To address this gap, we introduce PakBBQ, a culturally and regionally adapted extension of the original Bias Benchmark for Question Answering (BBQ) dataset. PakBBQ comprises over 214 templates, 17180 QA pairs across 8 categories in both English and Urdu, covering eight bias dimensions including age, disability, appearance, gender, socio-economic status, religious, regional affiliation, and language formality that are relevant in Pakistan. We evaluate multiple multilingual LLMs under both ambiguous and explicitly disambiguated contexts, as well as negative versus non negative question framings. Our experiments reveal (i) an average accuracy gain of 12\% with disambiguation, (ii) consistently stronger counter bias behaviors in Urdu than in English, and (iii) marked framing effects that reduce stereotypical responses when questions are posed negatively. These findings highlight the importance of contextualized benchmarks and simple prompt engineering strategies for bias mitigation in low resource settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。