构建多语言敏感问答数据集，揭示大模型伦理偏见的跨语言差异
Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models
- 构建涵盖17个敏感话题的多语言问答数据集MSQAD
- 统计检验显示多数语言间响应存在显著伦理偏见差异
- 公开数据集助力研究大模型跨语言偏见问题
尽管大型语言模型取得了显著进展，但研究表明这些系统中仍存在社会偏见。本文深入探究了大模型在涉及全球关注且可能敏感话题上的伦理偏见，并假设这些偏见可能源于语言差异。为此，我们构建了多语言敏感问答数据集（MSQAD），从人权观察组织覆盖17个主题的新闻文章中生成多语言的社会敏感问题及对应回答。通过两种统计假设检验，分析不同语言和话题下响应中的偏见。结果表明，在大多数情况下零假设被拒绝，说明跨语言差异导致了偏见。这表明伦理偏见在多种语言中普遍存在，甚至在不同大模型之间也广泛存在。通过公开提供所提出的MSQAD，我们旨在促进未来关于大模型跨语言偏见及其变体模型的研究。
原文摘要 · Abstract (English)
Despite the recent strides in large language models, studies have underscored the existence of social biases within these systems. In this paper, we delve into the validation and comparison of the ethical biases of LLMs concerning globally discussed and potentially sensitive topics, hypothesizing that these biases may arise from language-specific distinctions. Introducing the Multilingual Sensitive Questions & Answers Dataset (MSQAD), we collected news articles from Human Rights Watch covering 17 topics, and generated socially sensitive questions along with corresponding responses in multiple languages. We scrutinized the biases of these responses across languages and topics, employing two statistical hypothesis tests. The results showed that the null hypotheses were rejected in most cases, indicating biases arising from cross-language differences. It demonstrates that ethical biases in responses are widespread across various languages, and notably, these biases were prevalent even among different LLMs. By making the proposed MSQAD openly available, we aim to facilitate future research endeavors focused on examining cross-language biases in LLMs and their variant models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。