arXiv:2602.17283cs.CLcs.AI2026-02

首个跨语言价值观判断基准,评估大模型在多语种下对深层价值的判断能力。

Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspective

  • 采用人机协作双阶段框架,解决文化差异与学科复杂性难题。
  • 构建包含4750组问答的跨语言数据集,覆盖14种语言与7类全球议题。
  • 揭示主流大模型在跨语言价值观判断上的明显短板,适合评测与改进模型伦理能力。

随着大语言模型在全球范围的应用,现有评估范式主要关注多语言任务的事实性能,忽视了模型在跨语言情境下对内容深层价值观的判断能力。为填补这一空白,本文首先揭示构建价值观判断基准面临的两大挑战:文化多样性与学科复杂性,并提出一种新型两阶段人机协作标注框架,用于识别问题范畴与性质、建立具体标注标准,并利用多个大模型进行最终审核。基于此框架,本文推出首个跨语言价值观判断基准——X-Value,涵盖4,750个问题-答案对,覆盖14种语言和7大全球议题类别,并提供12项细粒度标注元数据,以支持对模型表现的严格评估。通过对17个大模型在不同提示策略下的系统性测评,多维度分析准确率与F1分数,揭示其在跨语言价值观判断中的普遍局限性,以及在不同议题和语言间的性能差异。该工作强调提升大模型深层价值观感知与判断能力的紧迫性。

原文摘要 · Abstract (English)

As large language models (LLMs) are employed worldwide, existing evaluation paradigms for their multilingual capabilities primarily focus on factual task performance, neglecting the ability to judge content's deep-level values across multiple languages. To bridge this gap, we first reveal two primary challenges in constructing values judgment benchmarks, cultural diversity and disciplinary complexity, and propose a novel two-stage human-AI collaborative annotation framework to alleviate them. This framework identifies the issue scope and nature, establishes specific annotation criteria, and utilizes multiple LLMs for final review. Building upon this framework, we introduce \textbf{X-Value}, the first \textit{Cross-lingual Values Judgment Benchmark} designed to evaluate the capability of LLMs in judging deep-level values of content. X-Value comprises 4,750 Question-Answer pairs across 14 languages, covering 7 major global issue categories, and provides 12 granular annotation metadata to facilitate a rigorous evaluation of model performance. Systematic evaluations of X-Value are conducted across 17 LLMs using distinct prompting strategies. Multi-dimensional analysis of accuracy and F1-scores reveals their limitations in cross-lingual values judgment and indicates performance disparities across categories and languages. This work highlights the urgent need to improve the underlying, values-aware content judgment capability of LLMs.\footnote{Samples of X-Value are available at https://huggingface.co/datasets/Whitolf/X-Value.}

价值观判断跨语言大模型评估数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。