用维基数据构建拉美社会文化偏见数据集，评估大模型在拉美语境下的知识差距。
Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America
- 结合维基百科、维基数据与社科专家知识，构建多语言拉美文化问答对。
- 创建超2.6万条西班牙语/葡萄牙语/英语多选题，揭示模型在拉美各国表现差异。
- 发现模型更熟悉伊比利亚西班牙文化，且原生语言下表现更好。
大型语言模型在不同文化背景下存在不平等现象，多数开源模型基于全球北方数据训练，对其他文化表现出偏见。尤其缺乏非英语语言、特别是拉丁美洲（Latam）地区的偏见检测资源，尽管该地区文化多样但共享一定文化基础。本文提出利用维基百科内容、维基数据知识图谱结构及社会科学专家知识，构建基于拉美各国流行与社会文化的问答对数据集。我们创建了包含超过26,000个问题和对应答案的LatamQA数据库，从26,000篇维基百科文章中提取并转换为西班牙语和葡萄牙语的多选题（MCQ），再翻译成英语。通过该数据集量化各类大模型的知识水平，发现：(i) 拉美各国间模型表现存在显著差异，部分国家问题更易被模型解答；(ii) 模型在其原生语言中表现更优；(iii) 伊比利亚西班牙文化比拉美整体文化更受模型掌握。
原文摘要 · Abstract (English)
Large Language Models (LLMs) exhibit inequalities with respect to various cultural contexts. Most prominent open-weights models are trained on Global North data and show prejudicial behavior towards other cultures. Moreover, there is a notable lack of resources to detect biases in non-English languages, especially from Latin America (Latam), a continent containing various cultures, even though they share a common cultural ground. We propose to leverage the content of Wikipedia, the structure of the Wikidata knowledge graph, and expert knowledge from social science in order to create a dataset of question/answer (Q/As) pairs, based on the different popular and social cultures of various Latin American countries. We create the LatamQA database of over 26k questions and associated answers extracted from 26k Wikipedia articles, and transformed into multiple-choice questions (MCQ) in Spanish and Portuguese, in turn translated to English. We use this MCQ to quantify the degree of knowledge of various LLMs and find out (i) a discrepancy in performances between the Latam countries, ones being easier than others for the majority of the models, (ii) that the models perform better in their original language, and (iii) that Iberian Spanish culture is better known than Latam one.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。