测试大模型在群聊中识别隐性规范并响应制裁的能力
SCENE: Recognizing Social Norms and Sanctioning in Group Chats
- 构建群聊场景模拟隐性规范与群体制裁行为
- Claude Opus 4.7和Gemini 3.1 Pro适应能力显著优于开源模型
- 适合研究大模型社会互动与动态评估的学者
在线群聊中存在未明说的行为规范，违反时会引发群体制裁。当前对大模型是否能识别并适应这些规范的研究仍不充分。我们提出SCENE，一个聚焦多角色对话中隐性规范与社会制裁的基准评测体系。SCENE通过设定有隐藏规范的角色脚本，生成可能触发违规的情境，并评估主体模型对制裁的响应及从同伴行为中学习规范的能力。我们采用两个行为评估指标：对负面制裁的响应能力，以及从同伴行为中适应新规范的能力。在该基准上评估了六种前沿及开源模型，结果表明Claude Opus 4.7与Gemini 3.1 Pro在适应隐性规范方面显著优于其他开放权重模型。SCENE为近期呼吁的大模型社会能力动态交互式评估提供了重要基准。
原文摘要 · Abstract (English)
Online group chats are social spaces with implicit behavior patterns that, when broken, are often met with social sanctioning from the group. The ability and willingness of LLM-based agents to recognize and adapt to these norms remains mostly unexplored. We introduce SCENE, a social-interaction benchmark focused on implicit norms and social sanctioning in multi-party chat. SCENE generates plausible non-roleplay scenarios with scripted personas that follow a hidden norm, create opportunities for the subject agent to violate it, and sanction breaches when they occur. We further propose behavioral evaluation metrics for two functional adaptation abilities: responsiveness to negative sanctioning, and adapting norm from peers behavior. We evaluate six frontier and open-weight models on SCENE. Our results show that Claude Opus 4.7 and Gemini 3.1 Pro adapt to implicit norms significantly more than the evaluated open-weight models. SCENE contributes one benchmark in the direction of recent calls for dynamic, interactional evaluation of LLM social capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。