测试大模型在心理健康领域的语用推理能力,发现部分模型表现优异。
P-ReMIS: Pragmatic Reasoning in Mental Health and a Social Implication
- 构建心理领域语用推理数据集PRiMH,设计蕴含与预设任务
- Mistral和Qwen在任务中表现突出,具备较强推理能力
- 对比三款先进模型对心理健康污名的回应,Claude-3.5-haiku更负责任
尽管可解释性与可理解性在人工智能与自然语言处理的心理健康应用中备受关注,但推理能力尚未得到同等重视。为弥合这一差距,我们研究了大语言模型(LLMs)在心理健康领域的语用推理能力。为此,我们构建了PRiMH数据集,并提出了包含语用蕴含与预设现象的心理健康语用推理任务,具体包括两项蕴含任务与一项预设任务。为评估该数据集与任务,我们采用四款模型:Llama3.1、Mistral、MentaLLaMa与Qwen。实验结果表明,Mistral与Qwen在该领域展现出显著的推理能力。随后,我们通过滚动注意力机制分析MentaLLaMA在这些任务中的行为表现。此外,我们还设计了三种StiPRompts,利用GPT4o-mini、Deepseek-chat与Claude-3.5-haiku三款先进模型,探究大模型对心理健康污名的态度。评估结果显示,Claude-3.5-haiku在应对污名问题时表现更为负责任。
原文摘要 · Abstract (English)
Although explainability and interpretability have received significant attention in artificial intelligence (AI) and natural language processing (NLP) for mental health, reasoning has not been examined in the same depth. Addressing this gap is essential to bridge NLP and mental health through interpretable and reasoning-capable AI systems. To this end, we investigate the pragmatic reasoning capability of large-language models (LLMs) in the mental health domain. We introduce PRiMH dataset, and propose pragmatic reasoning tasks in mental health with pragmatic implicature and presupposition phenomena. In particular, we formulate two tasks in implicature and one task in presupposition. To benchmark the dataset and the tasks presented, we consider four models: Llama3.1, Mistral, MentaLLaMa, and Qwen. The results of the experiments suggest that Mistral and Qwen show substantial reasoning abilities in the domain. Subsequently, we study the behavior of MentaLLaMA on the proposed reasoning tasks with the rollout attention mechanism. In addition, we also propose three StiPRompts to study the stigma around mental health with the state-of-the-art LLMs, GPT4o-mini, Deepseek-chat, and Claude-3.5-haiku. Our evaluated findings show that Claude-3.5-haiku deals with stigma more responsibly compared to the other two LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。