首个针对果阿地区身份群体的偏见评测基准，揭示多语言模型在地方文化上的认知缺陷。
AmchiBias: Measuring Stereotypical Bias in Goan Identity Groups with a Minimal Pair Dataset in English and Konkani
- 构建313组最小差异对数据集，覆盖8个社会人口维度的英、柯内克尼双语样本
- 多语言模型在柯内克尼语上接近随机得分，暴露其对果阿文化的理解缺失
- 英文查询下模型更倾向泛印度刻板印象，反映预训练信号而非真实地方知识
社会文化刻板偏见是自然语言处理系统开发与部署中的重要考量，但通常仅在国家层面被关注，忽视了丰富的次国家级社会文化结构。本文提出AmchiBias，首个针对印度果阿邦的本土文化偏见评测基准，涵盖多种果阿身份群体，包含313组最小差异对，覆盖8个社会人口维度，支持英语和德纳加里柯内克尼语。我们在此基准上评估了五种多语言编码器模型的刻板偏见表现。结果显示，在柯内克尼语中模型得分接近随机水平，表明通用多语言模型存在语言能力不足，而印度语种模型则缺乏果阿文化理解力。在英文查询下，具有更强印度语种覆盖的模型对泛印度群体表现出更高偏见，远高于对超本地果阿群体的偏见，说明英文信号反映的是泛印度预训练关联，而非真实的果阿文化知识。研究揭示了低资源多语言NLP在超本地社区身份评估上的关键缺口。
原文摘要 · Abstract (English)
Socio-cultural stereotypical bias is an important consideration in the development and deployment of NLP systems. It is however often considered only at the national level, despite rich subnational socio-cultural structures. We present AmchiBias, the first benchmark for measuring socio-cultural stereotypical bias for the Indian state of Goa with its unique historically multicultural setting. It covers various Goan identity groups and comprises 313 minimal pairs across eight sociodemographic dimensions in both English and Devanagari Konkani. We then evaluate stereotypical bias in five multilingual encoder models on this benchmark. We find near-chance scores in Konkani, reflecting language incompetence for general multilingual models and a lack of Goan cultural competence for Indian language models. Queried in English, models with a stronger Indian language coverage show higher bias for pan-Indian groups than hyperlocal Goan groups. This suggests the English signal reflects pan-Indian pretraining associations rather than genuine Goan cultural knowledge. Our findings highlight a critical gap in low-resource multilingual NLP evaluation for hyperlocal community identities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。