用结构化推理评估大模型角色一致性，更准更可信。
Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference
- 将角色一致性拆解为任务框架、人际立场、语言风格三维度
- 通过逆向推断提升评估准确性和稳定性，避免主观偏差
- 适合角色扮演、对话系统开发与评测的开发者
随着大语言模型被广泛用于模拟多样人类角色，确保角色一致性——即代理行为在心理和风格上持续反映目标角色特征——已成为关键要求。然而，现有评估方法主要依赖整体式LLM裁判，易产生‘整体判断幻觉’，或使用静态心理量表，无法捕捉动态对话中的情境依赖性一致性。为此，我们提出PRISM（基于逆向SFL建模的角色推理），一个基于心理语言学的框架，将角色一致性评估重构为结构化的逆向推理任务。受系统功能语言学（SFL）启发，PRISM将角色一致性分解为三个功能维度：任务框架、人际立场和语言风格。它在角色条件标签空间上估计各维度证据，并聚合为可解释、可审计的评估过程。实验表明，PRISM在准确性与稳定性上均优于传统整体评判，提供了更可靠的评估框架。
原文摘要 · Abstract (English)
As large language models are increasingly deployed to simulate diverse human characters, ensuring persona fidelity, defined as the extent to which an agent's behavior consistently reflects the psychological and stylistic characteristics of a target persona, has become a critical requirement. However, existing evaluation paradigms primarily rely on either holistic LLM-based judges, which are prone to "holistic appraisal hallucination'', or static psychometric inventories, which fail to capture the context-dependent fidelity required in dynamic dialogue. To address these limitations, we propose PRISM (Persona Reasoning with Inverse SFL-based Modeling), a psycholinguistically grounded framework that reformulates persona fidelity evaluation as a structured inverse inference task. Inspired by Systemic Functional Linguistics (SFL), PRISM decomposes persona fidelity into three functional dimensions: Task Framing, Interpersonal Stance, and Linguistic Style. It estimates dimension-specific evidence over a persona-conditioned label space and aggregates these signals into an interpretable and auditable evaluation process. Experiments show that PRISM yields more accurate and stable judgements than traditional holistic judging, providing a more reliable framework for persona fidelity evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。