首个考虑对话历史的LLM人格评估框架，发现上下文让回答更一致但可能改变人格表现。
CAPE: Context-Aware Personality Evaluation Framework for Large Language Models
- 引入对话历史构建上下文感知的人格评估框架，突破孤立答题局限。
- 实验显示上下文提升一致性，但GPT-3.5-Turbo等模型出现极端人格偏移。
- 适合研究LLM行为稳定性、人格建模及角色扮演应用的开发者与评测人员。
心理测量测试传统上用于评估人类行为特质，现被应用于大型语言模型（LLMs）以评估其行为特征。然而，现有研究采用无上下文的方法，孤立回答每个问题以避免上下文影响。我们称之为“迪士尼世界测试”，这是一种脱离真实应用场景的虚拟设定，忽略了对话历史对回应的实际影响。为弥合这一差距，我们提出了首个面向大语言模型的上下文感知人格评估（CAPE）框架，融合先前的对话交互。为全面分析上下文的影响，我们引入新指标量化模型响应的一致性，这是人类行为的基本特征。我们在7个LLM上进行广泛实验，结果表明，对话历史通过上下文学习提升了响应一致性，但也导致人格偏移，其中GPT-3.5-Turbo和GPT-4-Turbo表现出极端偏差。尽管GPT系列对问题顺序具有鲁棒性，但Gemini-1.5-Flash和Llama-8B则表现出显著敏感性。此外，GPT系列模型的回应既源于内在人格特质也受先前互动影响，而Gemini-1.5-Flash和Llama-8B则高度依赖先前互动。最后，将本框架应用于角色扮演代理（RPAs）时，发现基于上下文的人格变化能提高响应一致性，并更贴近人类判断。代码与数据集已公开于：https://github.com/jivnesh/CAPE
原文摘要 · Abstract (English)
Psychometric tests, traditionally used to assess humans, are now being applied to Large Language Models (LLMs) to evaluate their behavioral traits. However, existing studies follow a context-free approach, answering each question in isolation to avoid contextual influence. We term this the Disney World test, an artificial setting that ignores real-world applications, where conversational history shapes responses. To bridge this gap, we propose the first Context-Aware Personality Evaluation (CAPE) framework for LLMs, incorporating prior conversational interactions. To thoroughly analyze the influence of context, we introduce novel metrics to quantify the consistency of LLM responses, a fundamental trait in human behavior. Our exhaustive experiments on 7 LLMs reveal that conversational history enhances response consistency via in-context learning but also induces personality shifts, with GPT-3.5-Turbo and GPT-4-Turbo exhibiting extreme deviations. While GPT models are robust to question ordering, Gemini-1.5-Flash and Llama-8B display significant sensitivity. Moreover, GPT models response stem from their intrinsic personality traits as well as prior interactions, whereas Gemini-1.5-Flash and Llama--8B heavily depend on prior interactions. Finally, applying our framework to Role Playing Agents (RPAs) shows context-dependent personality shifts improve response consistency and better align with human judgments. Our code and datasets are publicly available at: https://github.com/jivnesh/CAPE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。