揭示多语言大模型隐含的全球与本地偏见，用跨语言测试发现美国中心倾向。
Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs
- 构建12语言的无线索问答集LocQA，检测模型对地域事实的隐性偏好。
- 32个模型中普遍出现以美国为基准的全球偏见，指令微调后更严重。
- 同一语言下模型倾向高人口地区，像人口概率计算器一样运作。
多语言大语言模型虽缩小了语言流畅度差距，却暴露于偏见风险中，因知识与规范可能跨语言传播。本文提出LocQA测试集，包含12种语言的2,156个不带地域线索的问答，涉及法律、日期、度量等依赖地域的事实。模型在不同语言下回答此类问题，可揭示其隐含的先验偏好。评估32个模型后，发现两类结构性偏见：跨语言上，即便非英语提问，模型仍倾向美国相关答案；且经指令微调的模型偏见更强。同语言内，当多个地区适用时，模型优先选择人口更多地区。这些发现有助于塑造模型的本地行为，并量化训练阶段对偏见的影响。
原文摘要 · Abstract (English)
Multilingual large language models (LLMs) have minimized the fluency gap between languages. This advancement, however, exposes models to the risk of biased behavior, as knowledge and norms may propagate across languages. In this work, we aim to quantify models' inter- and intra-lingual biases, via their ability to answer locale-ambiguous questions. To this end, we present LocQA, a test set containing 2,156 questions in 12 languages, referring to various locale-dependent facts such as laws, dates, and measurements. The questions do not contain indications of the locales they relate to, other than the querying language itself. LLMs' responses to LocQA locale-ambiguous questions thus reveal models' implicit priors. We used LocQA to evaluate 32 models, and detected two types of structural biases. Inter-lingually, we show a global bias towards answers relevant to the US-locale, even when models are asked in languages other than English. Moreover, we discovered that this global bias is exacerbated in models that underwent instruction tuning, compared to their base counterparts. Intra-lingually, we show that when multiple locales are relevant for the same language, models act as demographic probability engines, prioritizing locales with larger populations. Taken together, insights from LocQA may help in shaping LLMs' desired local behavior, and in quantifying the impact of various training phases on different kinds of biases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。