用对抗性测试评估大模型在心理咨询训练中的角色一致性
Evaluating Role-Consistency in LLMs for Counselor Training
- 构建对抗性数据集，检验大模型维持角色的能力
- 发现Vicuna在复杂对话中角色一致性显著下降
- 为心理训练场景提供可复用的LLM评估框架
在线心理咨询的兴起凸显了未来咨询师培训方法的有效性需求。本文拓展了VirCo（虚拟客户）研究，该工具旨在补充传统角色扮演教学法，通过模拟真实客户互动。基于前期工作，我们引入包含对抗性攻击的新数据集，用于测试大语言模型（LLMs）在对话中保持指定角色（角色一致性）的能力。研究聚焦于评估Vicuna模型在虚拟客户交互中的角色一致性和对话连贯性，并与先前研究结果进行对比。此外，我们还评估并比较了多种开源大模型在维持角色一致性方面的表现。主要贡献包括构建对抗性数据集、评估对话连贯性与人物一致性，以及提供不同大模型的对比分析。
原文摘要 · Abstract (English)
The rise of online counseling services has highlighted the need for effective training methods for future counselors. This paper extends research on VirCo, a Virtual Client for Online Counseling, designed to complement traditional role-playing methods in academic training by simulating realistic client interactions. Building on previous work, we introduce a new dataset incorporating adversarial attacks to test the ability of large language models (LLMs) to maintain their assigned roles (role-consistency). The study focuses on evaluating the role consistency and coherence of the Vicuna model's responses, comparing these findings with earlier research. Additionally, we assess and compare various open-source LLMs for their performance in sustaining role consistency during virtual client interactions. Our contributions include creating an adversarial dataset, evaluating conversation coherence and persona consistency, and providing a comparative analysis of different LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。