测试大模型对伊斯兰逊尼与什叶派的隐性偏见，发现语言和地理位置会改变其宗教立场。
SectEval: Evaluating the Latent Sectarian Preferences of Large Language Models
- 设计双语测试集SectEval，评估15个主流模型在宗教立场上的偏见。
- 英语中多数模型倾向什叶派，换为印地语后转为倾向逊尼派。
- 高级模型会随用户所在地调整答案，小型模型则固定偏向逊尼派。
随着大语言模型（LLMs）成为宗教知识的重要来源，了解其对不同群体是否公平至关重要。本研究首次衡量大模型对伊斯兰教两大派别——逊尼派与什叶派——的处理差异。我们提出了一个名为SectEval的测试，包含88个问题，涵盖英语与印地语两种语言，用于评估15个顶级大模型（包括专有与开源模型）的偏见。结果显示，语言显著影响模型输出：在英语中，DeepSeek-v3和GPT-4o等强大模型普遍倾向于什叶派答案；而在印地语中，这些模型转向支持逊尼派。这意味着仅因语言切换，用户就可能获得截然不同的宗教建议。此外，我们还考察了位置因素：Claude-3.5等先进模型会根据用户所在国家调整回答——伊朗用户获什叶派回应，沙特用户则得逊尼派回答。相比之下，较小的模型（尤其是印地语版本）无视用户位置，始终维持逊尼派立场。这表明，人工智能并非中立，其“宗教真相”会随语言和地理位置而变化。数据集已公开于https://github.com/secteval/SectEval/
原文摘要 · Abstract (English)
As Large Language Models (LLMs) becomes a popular source for religious knowledge, it is important to know if it treats different groups fairly. This study is the first to measure how LLMs handle the differences between the two main sects of Islam: Sunni and Shia. We present a test called SectEval, available in both English and Hindi, consisting of 88 questions, to check the bias-ness of 15 top LLM models, both proprietary and open-weights. Our results show a major inconsistency based on language. In English, many powerful models DeepSeek-v3 and GPT-4o often favored Shia answers. However, when asked the exact same questions in Hindi, these models switched to favoring Sunni answers. This means a user could get completely different religious advice just by changing languages. We also looked at how models react to location. Advanced models Claude-3.5 changed their answers to match the user's country-giving Shia answers to a user from Iran and Sunni answers to a user from Saudi Arabia. In contrast, smaller models (especially in Hindi) ignored the user's location and stuck to a Sunni viewpoint. These findings show that AI is not neutral; its religious ``truth'' changes depending on the language you speak and the country you claim to be from. The data set is available at https://github.com/secteval/SectEval/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。