现有心理量表无法准确预测大模型行为,生成式分析更可靠。
Human Psychometric Questionnaires Mischaracterize LLM Behavior
- 用问卷自评与真实对话生成对比,发现两法结果差异大
- 问卷中一致的得分在真实对话中消失,显示模型行为不稳
- 适合关注模型真实交互表现的研究者和开发者
我们检验了人类心理量表是否可作为可靠工具,用于刻画和预测大语言模型在日常用户交互中的行为。通过对比八种开源大模型在两种方法下的价值与人格画像:基于成熟量表(PVQ-40/21 和 BFI-44/10)的李克特自评,以及对日常用户查询中蕴含价值倾向回答的生成概率分析,发现两者存在显著分歧。同一构念内的题项一致性——常被视作模型稳定特质的证据——在生成概率中完全消失。我们归因于量表题干中明确的词汇线索使模型能识别目标构念并做出符合语境、社会期待的回应;而真实用户提问缺乏此类提示。此外,以人口统计角色提示引导模型回答问卷时,其反应符合真实人类模式,但在真实对话生成中并未出现相应变化,表明模型难以在实际交互中模拟特定人群行为。总体而言,研究显示人类心理量表不足以预测大模型行为,建议采用生成式画像作为更准确的评估方式。
原文摘要 · Abstract (English)
We examine whether human psychometric questionnaires can serve as reliable tools for characterizing and predicting LLM behavior in everyday user interactions. We analyze eight open-source LLMs by comparing their value and personality profiles derived from two different methods: Likert self-reports on established questionnaires (PVQ-40/21 and BFI-44/10) and generation probabilities over value-laden responses to everyday user queries. The two profiles diverge substantially. Within-construct item consistency, often cited as evidence of stable LLM dispositions, disappears in generation probabilities. We attribute this gap to the fact that explicit lexical cues in established questionnaire items allow models to recognize the target construct and respond in alignment-consistent, socially desirable ways, whereas realistic user queries provide no such cues. In addition, demographic persona prompts shift models' responses to human questionnaires in ways consistent with real human patterns, but no such shifts appear in the generation probabilities of responses to realistic user queries, showing their limited ability to simulate the behaviors of target demographics in real-world user interactions. Overall, our study shows that human psychometric questionnaires are insufficient tools for predicting LLM behavior and suggests generation-based profiling as a more accurate measure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。