测试大模型在角色冲突下的情境敏感性，发现多数模型更依赖固定偏好而非实时情境。
RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivity
- 构建13000+场景的基准，通过情境紧迫度控制变量
- 10个大模型中多数决策偏离客观情境，偏向角色固有偏好
- 适合研究模型社会推理能力或偏见评估的研究者
人们常面临角色冲突——多重社会角色期望相互矛盾且无法同时满足。随着大语言模型越来越多地参与此类社会互动，一个关键问题浮现：面对这类困境时，模型是优先响应动态情境线索，还是依赖已学习的角色偏好？为此，我们提出RoleConflictBench，一个用于衡量大模型在角色冲突情景中情境敏感性的新基准。为在主观领域实现客观评估，我们引入情境紧迫度作为决策约束。通过三阶段流程，在五个社会领域内生成超过13,000个真实场景，涵盖65种角色，并系统调节竞争情境的紧迫程度。这一受控设置使我们能够定量测量情境敏感性，判断模型决策是否与情境一致，或被角色偏好所主导。对10个大模型的分析显示，模型显著偏离客观基准。其决策主要受特定社会角色偏好支配，而非动态情境线索。
原文摘要 · Abstract (English)
People often encounter role conflicts -- social dilemmas where the expectations of multiple roles clash and cannot be simultaneously fulfilled. As large language models (LLMs) increasingly navigate these social dynamics, a critical research question emerges. When faced with such dilemmas, do LLMs prioritize dynamic contextual cues or the learned preferences? To address this, we introduce RoleConflictBench, a novel benchmark designed to measure the contextual sensitivity of LLMs in role conflict scenarios. To enable objective evaluation within this subjective domain, we employ situational urgency as a constraint for decision-making. We construct the dataset through a three-stage pipeline that generates over 13,000 realistic scenarios across 65 roles in five social domains by systematically varying the urgency of competing situations. This controlled setup enables us to quantitatively measure contextual sensitivity, determining whether model decisions align with the situational contexts or are overridden by the learned role preferences. Our analysis of 10 LLMs reveals that models substantially deviate from this objective baseline. Instead of responding to dynamic contextual cues, their decisions are predominantly governed by the preferences toward specific social roles.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。