测试大模型对印度区域性文化常识的识别能力,发现其普遍存在地域偏见。
Common to Whom? Regional Cultural Commonsense and LLM Bias in India
- 构建首个针对印度区域文化的常识评测集Indica,覆盖5个地区8个生活领域
- 仅39.4%问题在各地答案一致,大模型在区域题上准确率仅13.4%-20.9%
- 揭示模型偏好北中地区,严重低估东、西地区,适用于跨文化研究者
现有文化常识评测将国家视为单一整体,假设国内实践一致。但文化常识是否真在国家内部统一?我们提出Indica,首个用于检验该问题的基准,聚焦印度——一个拥有28个邦、8个中央直辖区和22种官方语言的多文化国家。我们在印度五个地区(北、南、东、西、中)收集了515个问题的人类标注答案,涵盖8个日常生活领域,共生成1630个区域特定问答对。惊人的是,仅有39.4%的问题在所有地区达成一致,表明印度的文化常识主要为区域性而非全国性。我们评估了八种先进大模型,发现两大关键缺陷:模型在区域特定问题上的准确率仅为13.4%-20.9%,且存在地理偏见——过度选择中北部地区作为默认选项(比预期高30-40%),而严重低估东部和西部地区。该方法论可推广至任何文化多元国家,涵盖基于人类学分类的问题设计、区域数据采集与偏见测量。
原文摘要 · Abstract (English)
Existing cultural commonsense benchmarks treat nations as monolithic, assuming uniform practices within national boundaries. But does cultural commonsense hold uniformly within a nation, or does it vary at the sub-national level? We introduce Indica, the first benchmark designed to test LLMs' ability to address this question, focusing on India - a nation of 28 states, 8 union territories, and 22 official languages. We collect human-annotated answers from five Indian regions (North, South, East, West, and Central) across 515 questions spanning 8 domains of everyday life, yielding 1,630 region-specific question-answer pairs. Strikingly, only 39.4% of questions elicit agreement across all five regions, demonstrating that cultural commonsense in India is predominantly regional, not national. We evaluate eight state-of-the-art LLMs and find two critical gaps: models achieve only 13.4%-20.9% accuracy on region-specific questions, and they exhibit geographic bias, over-selecting Central and North India as the "default" (selected 30-40% more often than expected) while under-representing East and West. Beyond India, our methodology provides a generalizable framework for evaluating cultural commonsense in any culturally heterogeneous nation, from question design grounded in anthropological taxonomy, to regional data collection, to bias measurement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。