提出新评估框架，测试大模型心理咨询师洞察用户隐性想法的能力
Can You Share Your Story? Modeling Clients' Metacognition and Openness for LLM Therapist Evaluation
- 设计可动态适应的虚拟来访者模拟器，隐藏真实心理状态
- 通过探索性提问覆盖度衡量模型对潜藏信念的理解程度
- 适合评估大模型在心理辅导中的共情与推理能力
理解来访者的内在想法和信念是心理咨询的核心，但现有LLM治疗师评估方法多依赖明确披露内部状态的模拟来访者，难以检验模型是否能发掘未明说的观点。为此，我们提出MindVoyager——一个可控且逼真的客户端模拟框架，能根据会话过程动态调整自身表现，构建更具挑战性的评估环境。同时引入新评价指标，通过测量大模型在咨询中对来访者信念与思维的探索深度，评估其认知洞察力。
原文摘要 · Abstract (English)
Understanding clients' thoughts and beliefs is fundamental in counseling, yet current evaluations of LLM therapists often fail to assess this ability. Existing evaluation methods rely on client simulators that clearly disclose internal states to the therapist, making it difficult to determine whether an LLM therapist can uncover unexpressed perspectives. To address this limitation, we introduce MindVoyager, a novel evaluation framework featuring a controllable and realistic client simulator which dynamically adapts itself based on the ongoing counseling session, offering a more realistic and challenging evaluation environment. We further introduce evaluation metrics that assess the exploration ability of LLM therapists by measuring their thorough understanding of client's beliefs and thoughts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。