arXiv:2505.17119cs.CLcs.LG2025-05被引 4

用大模型检测抑郁,发现其对隐含表达识别弱,优化后效果显著提升。

Systematic Evaluation of Machine-Generated Reasoning and PHQ-9 Labeling for Depression Detection Using Large Language Models

  • 拆解任务为子步骤,用指令策略系统分析模型推理过程
  • 在DepTweet数据集上,显性表达识别准确率更高,隐性表达仍存缺陷
  • 采用直接偏好优化(DPO)可有效提升性能并减少统计偏差

近期研究利用大语言模型(LLMs)进行早期心理健康检测,如抑郁症,常通过机器生成数据进行优化。然而,其检测能力可能存在未知弱点,且生成语料的质量控制仅依赖有限的人工验证。本文旨在系统评估LLM的推理能力并揭示潜在缺陷。首先,设计指令策略将检测任务分解为多个子任务,进行系统性分析;其次,构建对比性少样本与思维链提示,选取典型正负例;再者,对子任务进行人工标注并评估模型表现;最后,基于人类偏好的逻辑推理结果,探索优化策略。在DepTweet数据集上,针对三个子任务:1. 判断说话人是否描述自身抑郁;2. 准确检测PHQ-9症状;3. 最终判断是否抑郁,进行广泛对比。人工验证统计异常值显示,LLMs在分析显性抑郁语言时表现更优,而对隐性表达识别能力较弱。采用监督微调(SFT)和直接偏好优化(DPO)两种方法提升性能并降低统计偏差,其中DPO方法取得显著改进。

原文摘要 · Abstract (English)

Recent research leverages large language models (LLMs) for early mental health detection, such as depression, often optimized with machine-generated data. However, their detection may be subject to unknown weaknesses. Meanwhile, quality control has not been applied to these generated corpora besides limited human verifications. Our goal is to systematically evaluate LLM reasoning and reveal potential weaknesses. To this end, we first provide a systematic evaluation of the reasoning over machine-generated detection and interpretation. Then we use the models' reasoning abilities to explore mitigation strategies for enhanced performance. Specifically, we do the following: A. Design an LLM instruction strategy that allows for systematic analysis of the detection by breaking down the task into several subtasks. B. Design contrastive few-shot and chain-of-thought prompts by selecting typical positive and negative examples of detection reasoning. C. Perform human annotation for the subtasks identified in the first step and evaluate the performance. D. Identify human-preferred detection with desired logical reasoning from the few-shot generation and use them to explore different optimization strategies. We conducted extensive comparisons on the DepTweet dataset across the following subtasks: 1. identifying whether the speaker is describing their own depression; 2. accurately detecting the presence of PHQ-9 symptoms, and 3. finally, detecting depression. Human verification of statistical outliers shows that LLMs demonstrate greater accuracy in analyzing and detecting explicit language of depression as opposed to implicit expressions of depression. Two optimization methods are used for performance enhancement and reduction of the statistic bias: supervised fine-tuning (SFT) and direct preference optimization (DPO). Notably, the DPO approach achieves significant performance improvement.

抑郁症检测大模型推理DPO优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。