大模型在心理对话中,推理能力比专业微调更重要。
Reasoning Over Recall: Evaluating the Efficacy of Generalist Architectures vs. Specialized Fine-Tunes in RAG-Based Mental Health Dialogue Systems
- 用RAG框架对比通用模型与心理领域微调模型的性能
- 通用模型在共情上显著优于专业模型(3.72 vs. 3.26)
- 小模型靠推理胜出,适合需平衡安全与共情的场景
大型语言模型在心理咨询服务中面临幻觉和缺乏共情的双重挑战。尽管检索增强生成(RAG)可通过引用可信临床资料缓解幻觉问题,但何种模型更有效仍存疑问:是经过心理健康数据微调的专用模型,还是具备强大推理能力的通用模型?本文通过相同RAG管道(ChromaDB)测试四款开源模型:两款通用推理模型(Qwen2.5-3B、Phi-3-Mini)与两款领域专用微调模型(MentalHealthBot-7B、TherapyBot-7B)。采用LLM-as-a-Judge框架对50轮对话自动评估。结果表明,通用模型在共情方面显著优于专用模型(3.72 vs. 3.26,p < 0.001),尽管其规模更小(3B vs. 7B)。所有模型均表现安全,但通用模型具更强上下文理解力,且未出现专用模型的过拟合现象。总体而言,在基于RAG的心理咨询系统中,强推理能力比领域词汇训练更具价值——只要答案依托临床证据,优秀推理的通用模型能提供更共情、更平衡的支持。
原文摘要 · Abstract (English)
The deployment of Large Language Models (LLMs) in mental health counseling faces the dual challenges of hallucinations and lack of empathy. While the former may be mitigated by RAG (retrieval-augmented generation) by anchoring answers in trusted clinical sources, there remains an open question as to whether the most effective model under this paradigm would be one that is fine-tuned on mental health data, or a more general and powerful model that succeeds purely on the basis of reasoning. In this paper, we perform a direct comparison by running four open-source models through the same RAG pipeline using ChromaDB: two generalist reasoners (Qwen2.5-3B and Phi-3-Mini) and two domain-specific fine-tunes (MentalHealthBot-7B and TherapyBot-7B). We use an LLM-as-a-Judge framework to automate evaluation over 50 turns. We find a clear trend: the generalist models outperform the domain-specific ones in empathy (3.72 vs. 3.26, $p < 0.001$) in spite of being much smaller (3B vs. 7B), and all models perform well in terms of safety, but the generalist models show better contextual understanding and are less prone to overfitting as we observe in the domain-specific models. Overall, our results indicate that for RAG-based therapy systems, strong reasoning is more important than training on mental health-specific vocabulary; i.e. a well-reasoned general model would provide more empathetic and balanced support than a larger narrowly fine-tuned model, so long as the answer is already grounded in clinical evidence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。