大模型在真实临床对话中表现堪比人类，能有效开展动机访谈。
Benchmarking Motivational Interviewing Competence of Large Language Models
- 用MITI框架评估10个大模型的动机访谈能力，对比人类医生。
- 部分模型在反射性回应和问答比例上优于人类专家。
- 医生识别模型与真人对话准确率仅56%，适合资源匮乏场景。
动机访谈（MI）有助于物质滥用障碍的行为改变，其有效性通过动机访谈治疗完整性（MITI）框架衡量。尽管大语言模型（LLMs）可能生成符合MI规范的治疗师回应，但其在真实临床转录文本中的胜任力尚不明确。本研究旨在评估专有及开源模型在真实临床转录文本中的MI胜任力，并检验其与人类治疗师的可区分性。方法：从LMArena选取3个专有模型和7个开源模型，使用MITI 4.2框架在两个数据集上评估（96条人工构建的模型转录文本，34条真实临床转录文本）。保持客户端回应不变，迭代生成模型与人类治疗师的对话语句，通过包含MITI分项与冗长度的综合排名系统进行评分。两名独立精神科医生参与可区分性实验，以判断语句来源。结果：所有10个测试模型在MITI各维度均达到良好（全局分>3.5）至优秀（全局分>4）水平；其中三个表现最佳模型（gemma-3-27b-it、gemini-2.5-pro、grok-3）在真实转录文本中表现良好，复杂反射率（39% vs 96%）与反射-提问比（1.2 vs >2.8）均优于人类专家。可区分性实验中，医生识别准确率为56%，d-prime值分别为0.17（gemini-2.5-pro）和0.25（gemma-3-27b-it）。结论：大模型可在真实临床语境下实现良好水平的动机访谈能力。这些发现表明，即使开源模型也具备在低资源环境下扩展动机访谈服务的潜力。
原文摘要 · Abstract (English)
Motivational interviewing (MI) promotes behavioural change in substance use disorders. Its fidelity is measured using the Motivational Interviewing Treatment Integrity (MITI) framework. While large language models (LLMs) can potentially generate MI-consistent therapist responses, their competence using MITI is not well-researched, especially in real world clinical transcripts. We aim to benchmark MI competence of proprietary and open-source models compared to human therapists in real-world transcripts and assess distinguishability from human therapists. Methods: We shortlisted 3 proprietary and 7 open-source LLMs from LMArena, evaluated performance using MITI 4.2 framework on two datasets (96 handcrafted model transcripts, 34 real-world clinical transcripts). We generated parallel LLM-therapist utterances iteratively for each transcript while keeping client responses static, and ranked performance using a composite ranking system with MITI components and verbosity. We conducted a distinguishability experiment with two independent psychiatrists to identify human-vs-LLM responses. Results: All 10 tested LLMs had fair (MITI global scores >3.5) to good (MITI global scores >4) competence across MITI measures, and three best-performing models (gemma-3-27b-it, gemini-2.5-pro, grok-3) were tested on real-world transcripts. All showed good competence, with LLMs outperforming human-expert in Complex Reflection percentage (39% vs 96%) and Reflection-Question ratio (1.2 vs >2.8). In the distinguishability experiment, psychiatrists identified LLM responses with only 56% accuracy, with d-prime: 0.17 and 0.25 for gemini-2.5-pro and gemma-3-27b-it respectively. Conclusion: LLMs can achieve good MI proficiency in real-world clinical transcripts using MITI framework. These findings suggest that even open-source LLMs are viable candidates for expanding MI counselling sessions in low-resource settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。