arXiv:2509.13244cs.CL2025-09被引 5

测试大模型在真实访谈中推断人格特质的能力,发现表现普遍较差。

Evaluating LLM Alignment on Personality Inference from Real-World Interview Data

  • 用真实访谈数据和连续人格评分构建新评测基准
  • 所有模型预测与真实人格相关性低于0.26,效果有限
  • 显式推理对提升效果帮助不大,需改进模型对语义的理解

大型语言模型(LLMs)正被应用于需要细致心理理解的场景,如情感支持、心理咨询和决策辅助。然而,其在自然对话中识别人格特质的能力尚未充分探索。以往研究多基于社交媒体数据模拟模型人格,但缺乏对真实交互中连续人格评估的验证。为此,我们构建了一个包含半结构化访谈记录与经验证的连续五大性格特质评分的新基准。通过该数据集,系统评估了三种范式:(1) GPT-4.1 Mini 的零样本与思维链提示;(2) 对 RoBERTa 与 Meta-LLaMA 进行 LoRA 微调;(3) 使用预训练 BERT 和 OpenAI text-embedding-3-small 的静态嵌入进行回归。结果显示,所有模型预测与真实人格之间的皮尔逊相关系数均低于 0.26,表明当前 LLM 与已验证的心理构念对齐程度极低。思维链提示相比零样本仅带来微弱提升,说明人格推断更依赖潜在语义表示而非显式推理。这些发现揭示了将 LLM 与复杂人类特质对齐的挑战,并推动未来在特质特定提示、上下文感知建模和对齐导向微调方面的工作。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are increasingly deployed in roles requiring nuanced psychological understanding, such as emotional support agents, counselors, and decision-making assistants. However, their ability to interpret human personality traits, a critical aspect of such applications, remains unexplored, particularly in ecologically valid conversational settings. While prior work has simulated LLM "personas" using discrete Big Five labels on social media data, the alignment of LLMs with continuous, ground-truth personality assessments derived from natural interactions is largely unexamined. To address this gap, we introduce a novel benchmark comprising semi-structured interview transcripts paired with validated continuous Big Five trait scores. Using this dataset, we systematically evaluate LLM performance across three paradigms: (1) zero-shot and chain-of-thought prompting with GPT-4.1 Mini, (2) LoRA-based fine-tuning applied to both RoBERTa and Meta-LLaMA architectures, and (3) regression using static embeddings from pretrained BERT and OpenAI's text-embedding-3-small. Our results reveal that all Pearson correlations between model predictions and ground-truth personality traits remain below 0.26, highlighting the limited alignment of current LLMs with validated psychological constructs. Chain-of-thought prompting offers minimal gains over zero-shot, suggesting that personality inference relies more on latent semantic representation than explicit reasoning. These findings underscore the challenges of aligning LLMs with complex human attributes and motivate future work on trait-specific prompting, context-aware modeling, and alignment-oriented fine-tuning.

人格推断大模型评测心理学对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。