首个多语言文化评估框架，揭示大模型在跨文化理解中的偏见与差距
MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs
- 动态生成文化相关问题，结合反事实重述实现因果分析
- 覆盖13国13语种，提供近6万条文化认知与偏见数据实例
- 发现英语表现好不等于全球公平，语言-文化匹配影响关键
大型语言模型在服务多元全球用户时存在文化偏见且跨文化理解能力有限。我们提出MCEval，一种新型多语言评估框架，通过动态文化问题构建，并利用反事实重述和混淆因素重述实现因果分析。评估覆盖13种文化和13种语言，系统性地检验不同语言场景下的文化意识与文化偏见。框架共提供39,897个文化认知实例和17,940个文化偏见实例。实验结果揭示不同语言场景下性能差异显著，最优文化表现不仅与训练数据分布有关，还与语言-文化匹配度相关。评估结果还暴露公平性问题：在英语场景表现良好的方法，在其他语言中可能造成显著劣势。MCEval是首个全面的多语言文化评估框架，为深入理解大模型的文化理解能力提供了新视角。
原文摘要 · Abstract (English)
Large language models exhibit cultural biases and limited cross-cultural understanding capabilities, particularly when serving diverse global user populations. We propose MCEval, a novel multilingual evaluation framework that employs dynamic cultural question construction and enables causal analysis through Counterfactual Rephrasing and Confounder Rephrasing. Our comprehensive evaluation spans 13 cultures and 13 languages, systematically assessing both cultural awareness and cultural bias across different linguistic scenarios. The framework provides 39,897 cultural awareness instances and 17,940 cultural bias instances. Experimental results reveal performance disparities across different linguistic scenarios, demonstrating that optimal cultural performance is not only linked to training data distribution, but also is related to language-culture alignment. The evaluation results also expose the fairness issue, where approaches appearing successful in the English scenario create substantial disadvantages. MCEval represents the first comprehensive multilingual cultural evaluation framework that provides deeper insights into LLMs' cultural understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。