首次评估大模型心理辅导能力，发现中文表现优于英文。
Psychological Counseling Ability of Large Language Models
- 用心理咨询考试题测试主流大模型，分知识、分析、应用三类
- 中文题正确率最高46.5%（GLM-3），英文题最高43.9%（ERNIE-3.5）
- 引入指导手册后，模型正确率提升至59.6%，适合研究者参考
随着科技发展与人工智能进步，大型语言模型（LLMs）已在多个领域广泛应用。然而，在心理辅导领域，其能力尚未得到系统评估。本研究采用1096道来自中国国家三级心理咨询师考试的题目，涵盖知识型、分析型和应用型三类，评估主流大模型的心理辅导能力。结果显示，中文题中正确率由高到低为：GLM-3（46.5%）、GPT-4（46.1%）、ERNIE-3.5（45.7%）、Gemini（45.0%）、GPT-3.5（32.9%）；英文题中为：ERNIE-3.5（43.9%）、GPT-4（40.6%）、Gemini（36.6%）、GLM-3（29.9%）、GPT-3.5（29.5%）。卡方检验显示，各模型在中英文题目上的表现存在显著差异。进一步以《心理咨询师指南（三级）》为参考对ERNIE-3.5进行优化，正确率提升至59.6%，较初始的45.8%提高13.8%。研究首次系统评估了大模型的心理辅导能力，为未来改进提供依据。
原文摘要 · Abstract (English)
With the development of science and the continuous progress of artificial intelligence technology, Large Language Models (LLMs) have begun to be widely utilized across various fields. However, in the field of psychological counseling, the ability of LLMs have not been systematically assessed. In this study, we assessed the psychological counseling ability of mainstream LLMs using 1096 psychological counseling skill questions which were selected from the Chinese National Counselor Level 3 Examination, including Knowledge-based, Analytical-based, and Application-based question types. The analysis showed that the correctness rates of the LLMs for Chinese questions, in descending order, were GLM-3 (46.5%), GPT-4 (46.1%), Gemini (45.0%), ERNIE-3.5 (45.7%) and GPT-3.5 (32.9%). The correctness rates of the LLMs for English questions, in descending order, were ERNIE-3.5 (43.9%), GPT-4 (40.6%), Gemini (36.6%), GLM-3 (29.9%) and GPT-3.5 (29.5%). A chi-square test indicated significant differences in the LLMs' performance on Chinese and English questions. Furthermore, we subsequently utilized the Counselor's Guidebook (Level 3) as a reference for ERNIE-3.5, resulting in a new correctness rate of 59.6%, a 13.8% improvement over its initial rate of 45.8%. In conclusion, the study assessed the psychological counseling ability of LLMs for the first time, which may provide insights for future enhancement and improvement of psychological counseling ability of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。