评测大模型对印尼核心价值观的把握能力，发现普遍不足。
Pancasila-Dilemmas: Evaluating Large Language Models on Indonesian Human Value Dilemmas Grounded in Pancasila
- 构建1834个基于印尼国情的价值困境题集，覆盖五大民族价值。
- 所有模型在关键指标上均低于0.5，宗教与团结类难题最差。
- 适合研究多国文化适配或本土化大模型的学者使用。
大语言模型的价值对齐对确保其回应符合人类意图至关重要，但现有评估多聚焦西方或普世价值，缺乏针对特定国家价值体系的测评。本文提出Pancasila-Dilemmas数据集，包含1,834道源自印尼新闻的困境问题，按印尼国父哲学五大价值分类：宗教、人性、团结、民主与社会正义。数据经母语者校对，并由5位不同背景的印尼公民作答。我们评估了50个开源与闭源大模型，结果表明所有模型的概率匹配分数（PMS）均低于0.5，最大投票一致率（MVAS）为0.72。按价值维度分析，模型在宗教与团结相关困境中表现最差，暴露出对印尼本土价值观理解的显著差距。该数据集已公开于https://github.com/tjunlp-lab/Pancasila-Dilemmas。
原文摘要 · Abstract (English)
The value alignment of large language models (LLMs) is crucial for ensuring responses align with human intention and value preferences. However, most evaluations of value alignment focus on Western or universal values, while assessments grounded in the value systems of specific countries remain scarce. In this paper, we introduce Pancasila-Dilemmas, an evaluation dataset of 1,834 questions derived from Indonesian news, classified by 5 values of Pancasila: Religion, Humanity, Unity, Democracy, and Social Justice. This dataset reflects daily life in Indonesia, making it suitable for measuring the value alignment of LLMs deployed for Indonesia. To ensure a more rigorous evaluation, we choose scenarios containing dilemmas. The dataset is proofread by native speakers and answered by 5 diverse Indonesian citizens. We evaluate 50 closed- and open-source LLMs on our dataset. Results reveal that all evaluated LLMs achieves less than 0.5 Probability Match Score (PMS) and 0.72 Max-Vote Agreement Score (MVAS). Compared by each values, LLMs mostly struggle in Religion and Unity dilemma cases. This highlights a significant gap in capturing Indonesian values. The dataset is publicly available at https://github.com/tjunlp-lab/Pancasila-Dilemmas.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。