测试大模型代理在不同情境下行为是否一致,发现其看似像人却并不稳定。
Are LLM Agents Behaviorally Coherent? Latent Profiles for Social Simulation
- 设计问卷与对话实验,揭示代理的潜在行为特征
- 发现不同模型家族和大小间存在显著行为不一致
- 适合关注大模型可信性与实验替代性的研究者
大型语言模型(LLMs)的强大能力使其有望替代真实参与者进行人类实验研究。现有研究多聚焦于模型生成的问卷回答是否与人类一致,而本文提出更根本的问题:代理在不同实验设置下是否保持行为一致性?为此,我们设计了两项任务:(a) 通过一组问题揭示代理的潜在行为特征;(b) 在与其他代理的对话中检验其行为一致性。基于此,我们验证了多个行为假设,评估代理对话表现是否与其揭示的状态相符。结果显示,不同模型家族及模型规模之间存在显著的行为不一致。最重要的是,尽管代理可能生成与人类相似的回答,但其行为缺乏实证一致性,暴露出其在替代真实参与者方面存在关键缺陷。
原文摘要 · Abstract (English)
The impressive capabilities of Large Language Models (LLMs) raise the possibility that synthetic agents can serve as substitutes for real participants in human-subject research. To evaluate this claim, prior research has largely focused on whether LLM-generated survey responses align with those produced by human respondents whom the LLMs are prompted to represent. In contrast, we address a more fundamental question: Do agents maintain empirical consistency; aligning to human behavioral models when examined under different experimental settings? To this end, we develop a study designed to (a) ask a set of questions which reveals an agent's latent profile and (b) examine agent behavioral consistency in a conversational setting with other agents. This design enables us to explore a set of behavioral hypotheses to assess whether an agent's conversational behavior is consistent with what we would expect from its revealed state. Our findings show significant inconsistencies in LLMs across model families and at differing model sizes. Most importantly, we find that, although agents may generate responses matching those of their human counterparts, they fail to be empirically consistent, representing a critical gap in their capabilities to accurately substitute for real participants in human-subject research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。