arXiv:2605.23420cs.CL2026-05

提出自然对话中社会规范对齐的测量框架,支持人机、机机对话对比。

Naturalistic measure of social norms alignment

论文配图:Naturalistic measure of social norms alignment
图 1 · 摘自论文原文
  • 通过自由回答匹配度测量社会规范对齐,无需预设选项。
  • 在3000个丹麦社会困境上测试,模型排名稳定,邻里冲突对齐度更高。
  • 适合研究文化语境下的开放对话与大模型社会推理能力。

社会规范反映人们对可接受行为的共同期待。现有方法多依赖人工设计的封闭式评估(如选择题或预设陈述的一致性判断)。本文将社会规范对齐定义为不同解决方案在社会问题或困境中的共识程度。提出一种在自然、开放形式下测量对齐的框架,可通过解题匹配实现任意两个回应间的对齐评估,如人类与大模型、大模型与大模型、人类与人类之间。引入两种指标:陈述一致准确率和显式同意准确率,并构建包含3000个非平凡社会困境的丹麦语数据集。所有困境由三位文化背景固定的评审员提供参考解答。在模拟自然人机对话的交互设置中,评估多个大模型与人类回答的一致性。结果表明,所提指标能产生一致的模型排序,且不同困境类型间存在差异,如邻里矛盾和共居情境的对齐度更高。本工作为研究文化根基的社会推理提供了数据集与评估框架。

原文摘要 · Abstract (English)

Social norms reflect shared expectations on acceptable behavior. Measuring social norms alignment remains challenging, with existing approaches typically relying on artificial closed-form evaluations such as multiple-choice questionnaires or measuring agreement with predefined statements. In the context of this work, social norms alignment refers to measuring an agreement between solutions with respect to the social problem or dilemma. We propose a framework for measuring social norm alignment in naturalistic, free-form settings through solution matching. The framework enables us to measure alignment between any two dilemma responses e.g., LLMs to a human, LLMs to LLMs, or human to human. We introduce two metrics: stated and explicit agreement accuracy, and construct a dataset of 3k non-trivial social dilemmas in Danish. All dilemmas are assigned reference solutions derived from three panelists, who serve as culturally grounded judges. We evaluate the agreement of several LLMs and human responses in an interaction setup that resembles natural user-model conversations. Our results show that the proposed metrics produce consistent model rankings and reveal variation in agreement across different types of dilemmas, with higher agreement observed for topics such as neighbor conflicts and shared living situations. Overall, our work introduces a dataset and evaluation framework for studying culturally grounded social reasoning in naturalistic open-ended conversations.

社会规范大模型评估开放对话文化基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。