检测对话模拟中缺失的关键人格特征，提升用户模拟真实性
Is He Extroverted? Identifying Missing Relevant Personas for Faithful User Simulation
- 提出基于上下文选择题的评测框架，识别对话中缺失的人格维度
- 在电视剧猜谜数据集上验证，大模型可有效识别缺失人格特征
- 揭示大模型与人类在人格推理上的认知差异，适合对话系统研究者
现有用户模拟方法通常假设给定人格足以生成类人回复，但未验证关键人格是否缺失，影响模拟有效性。本文研究识别对话上下文中缺失但相关的人格维度（如‘是否价格敏感’）的任务。构建了PICQ-drama数据集（基于TVShowGuess），包含需依赖缺失人格才能正确回答的上下文感知选择题，并标注了导致用户选择模糊的缺失人格维度。设计多种评估标准进行测评。在该数据集上对主流大模型的测试表明该任务可行。跨标准评估与深入分析揭示了大模型与人类在人格推理上的认知差异，凸显不同人格类别在塑造回复中的独特作用。
原文摘要 · Abstract (English)
Existing user simulation approaches focus on generating user-like responses in dialogue. They often assume that the provided persona is sufficient for producing such responses, without verifying whether critical personas are supplied. This raises concerns about the validity of simulation results. To address this issue, we study the task of identifying persona dimensions (e.g., "whether the user is price-sensitive") that are relevant but missing in simulating a user's reply for a given dialogue context. We introduce PICQ-drama (constructed from TVShowGuess), a benchmark of context-aware choice questions, annotated with missing persona dimensions whose absence leads to ambiguous user choices. We further design diverse evaluation criteria for missing persona identification. Benchmarking leading LLMs on our PICQ-drama dataset demonstrates the feasibility of this task. Evaluation across diverse criteria, along with further analyses, reveals cognitive differences between LLMs and humans and highlights the distinct roles of different persona categories in shaping responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。