LLMs偏爱日本文化，语言资源越少越明显
Why are all LLMs Obsessed with Japanese Culture? On the Hidden Cultural and Regional Biases of LLMs
- 构建24语言文化问答数据集CROQ，评估模型文化偏好
- 英文等高资源语言下答案更多元，低资源语言倾向官方语国家
- 文化偏见在微调阶段出现，非预训练时期
大语言模型在文化覆盖与能力方面存在局限，某些情况下表现出特定文化偏见。尽管已有研究探讨过模型的文化能力，但尚未专门分析其在通用文化问题上的区域偏好。本文提出一个基于全面文化相关开放问题分类的新型数据集CROQ，包含24种语言的问题，并通过提示模型回答问题并提供样本位置进行评估。结果表明，与以往文化偏见研究相反，模型在回答中明显倾向日本等国家。此外，使用英语等高资源语言提示时，模型输出更具多样性；而低资源语言则更倾向于指向输入语言的官方国家。最后，我们还探究了文化偏见出现的时间点，结果显示首次明显迹象出现在监督微调阶段，而非预训练阶段。数据集已公开于https://huggingface.co/datasets/HiTZ/CROQ。
原文摘要 · Abstract (English)
LLMs have limitations when it comes to cultural coverage and competence, and in some cases, show specific cultural biases. Although prior studies have examined the cultural capabilities of LLMs, none have specifically investigated their regional preferences in generic culture-related questions. In this work, we propose a new dataset based on a comprehensive taxonomy of Culture-Related Open Questions (CROQ), with questions available in 24 languages. We evaluate LLMs by prompting them to answer questions from CROQ and provide a sample location. The results show that, contrary to previous cultural bias work, LLMs show a clear tendency towards countries such as Japan in their answers. Moreover, our results show that when prompting in languages such as English or other high-resource ones, LLMs tend to provide more diverse outputs. Low-resource languages, on the other hand, show more inclinations towards answering questions highlighting countries for which the input language is an official language. Finally, we also investigate at which point of LLM training this cultural bias emerges, with our results suggesting that the first clear signs appear after supervised fine-tuning, and not during pre-training. Dataset available at https://huggingface.co/datasets/HiTZ/CROQ
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。