测试大模型对加纳与美国的文化常识理解差异,发现模型更倾向美国观点。
Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the U.S
- 通过多轮调查构建525道跨文化常识题
- 模型在美式语境下表现优于加纳语境
- 揭示现有模型缺乏文化适应性,适合关注AI公平性的研究者
近期研究指出常识知识具有文化依赖性。本文提出AMAMMER$ε$,一个包含525道多项选择题的测试集,用于评估英语大模型在加纳与美国文化背景下的常识理解能力。通过三轮调查,分别邀请加纳和美国参与者撰写、评分并选择正确答案,确保题目设计与验证充分反映两地文化视角。最终结果表明,所有评测的开源英语大模型均更偏好美国标注者的答案;当题目明确指定文化背景时,模型具备一定适应能力,但在美国语境下性能始终更优。随着大量资源投入英语大模型发展,本研究强调需建立更具文化适应性的模型与评估体系,以满足全球多样化英语使用者的需求。
原文摘要 · Abstract (English)
Recent work has highlighted the culturally-contingent nature of commonsense knowledge. We introduce AMAMMER$ε$, a test set of 525 multiple-choice questions designed to evaluate the commonsense knowledge of English LLMs, relative to the cultural contexts of Ghana and the United States. To create AMAMMER$ε$, we select a set of multiple-choice questions (MCQs) from existing commonsense datasets and rewrite them in a multi-stage process involving surveys of Ghanaian and U.S. participants. In three rounds of surveys, participants from both pools are solicited to (1) write correct and incorrect answer choices, (2) rate individual answer choices on a 5-point Likert scale, and (3) select the best answer choice from the newly-constructed MCQ items, in a final validation step. By engaging participants at multiple stages, our procedure ensures that participant perspectives are incorporated both in the creation and validation of test items, resulting in high levels of agreement within each pool. We evaluate several off-the-shelf English LLMs on AMAMMER$ε$. Uniformly, models prefer answers choices that align with the preferences of U.S. annotators over Ghanaian annotators. Additionally, when test items specify a cultural context (Ghana or the U.S.), models exhibit some ability to adapt, but performance is consistently better in U.S. contexts than Ghanaian. As large resources are devoted to the advancement of English LLMs, our findings underscore the need for culturally adaptable models and evaluations to meet the needs of diverse English-speaking populations around the world.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。