arXiv:2411.03665cs.CLcs.AI2024-11EMNLP被引 11

用多元框架评测大模型道德观,发现中英文模型倾向不同

Evaluating Moral Beliefs across LLMs through a Pluralistic Framework

  • 构建472个中文道德情景数据集,通过决策排序分析模型道德偏好
  • 英文模型贴近中国大学生道德选择,更坚持个体主义;中文模型偏向集体主义且立场模糊
  • 发现所有模型均存在性别偏见,方法可跨文化比较智能体与人类道德

准确的道德信念对语言模型至关重要,但评估其道德观仍具挑战。本研究提出一个三模块框架,评估四种主流大模型的道德信念。首先,基于道德词汇构建包含472个道德抉择场景的中文数据集,通过模型在这些场景中的决策过程揭示其道德原则偏好,并通过排序识别不同模型间的道德信念差异。其次,通过道德辩论考察模型对其选择的坚定程度。结果显示,英文模型(ChatGPT、Gemini)在决策上接近中国大学生样本,表现出强坚持性及个体主义倾向;而中文模型(Ernie、ChatGLM)则偏向集体主义,道德选择与辩论中表现模糊。此外,所有被测模型均存在性别偏见。该方法为评估人工智能与人类智能的道德信念提供了新路径,支持跨文化道德价值比较。

原文摘要 · Abstract (English)

Proper moral beliefs are fundamental for language models, yet assessing these beliefs poses a significant challenge. This study introduces a novel three-module framework to evaluate the moral beliefs of four prominent large language models. Initially, we constructed a dataset containing 472 moral choice scenarios in Chinese, derived from moral words. The decision-making process of the models in these scenarios reveals their moral principle preferences. By ranking these moral choices, we discern the varying moral beliefs held by different language models. Additionally, through moral debates, we investigate the firmness of these models to their moral choices. Our findings indicate that English language models, namely ChatGPT and Gemini, closely mirror moral decisions of the sample of Chinese university students, demonstrating strong adherence to their choices and a preference for individualistic moral beliefs. In contrast, Chinese models such as Ernie and ChatGLM lean towards collectivist moral beliefs, exhibiting ambiguity in their moral choices and debates. This study also uncovers gender bias embedded within the moral beliefs of all examined language models. Our methodology offers an innovative means to assess moral beliefs in both artificial and human intelligence, facilitating a comparison of moral values across different cultures.

道德评估大模型文化差异偏见检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。