arXiv:2410.04501cs.CLcs.AI2024-10中稿 · IEEE International…被引 14

用大模型自动生成标签,提升社交媒体自杀内容识别准确率。

Leveraging Large Language Models for Suicide Detection on Social Media with Limited Labels

  • 用大模型生成伪标签,结合微调提升标注质量。
  • 集成多个模型使加权F1达0.770(公开集),比单模型高5个百分点。
  • 适合做心理健康预警的AI研究者和平台安全团队参考。

自杀念头的频发凸显了早期识别与干预的重要性。社交平台用户常在上面分享个人经历并寻求帮助,是发现高风险个体的重要渠道。但每日海量帖子令人工审核难以实现。本文探索利用大语言模型(LLMs)自动检测文本社交平台中的自杀相关内容。提出一种通过提示词生成未标注数据伪标签的新方法,并结合传统分类微调技术以提升标签准确性。为构建强性能检测模型,采用集成策略:使用Qwen2-72B-Instruct进行提示,融合Llama3-8B、Llama3.1-8B和Gemma2-9B等微调模型。在IEEE Big Data 2024 Big Data Cup的“社交媒体自杀意念检测”挑战赛数据集上评估。实验表明,集成模型显著提升检测准确率,较单模型提升5个百分点,在公开测试集上达到加权F1 0.770,私有测试集为0.731,为识别社交媒体自杀内容提供有效方案。分析显示,模型选择影响提示效果,更大模型表现更优。代码与检查点已公开于https://github.com/khanhvynguyen/Suicide_Detection_LLMs。

原文摘要 · Abstract (English)

The increasing frequency of suicidal thoughts highlights the importance of early detection and intervention. Social media platforms, where users often share personal experiences and seek help, could be utilized to identify individuals at risk. However, the large volume of daily posts makes manual review impractical. This paper explores the use of Large Language Models (LLMs) to automatically detect suicidal content in text-based social media posts. We propose a novel method for generating pseudo-labels for unlabeled data by prompting LLMs, along with traditional classification fine-tuning techniques to enhance label accuracy. To create a strong suicide detection model, we develop an ensemble approach involving prompting with Qwen2-72B-Instruct, and using fine-tuned models such as Llama3-8B, Llama3.1-8B, and Gemma2-9B. We evaluate our approach on the dataset of the Suicide Ideation Detection on Social Media Challenge, a track of the IEEE Big Data 2024 Big Data Cup. Additionally, we conduct a comprehensive analysis to assess the impact of different models and fine-tuning strategies on detection performance. Experimental results show that the ensemble model significantly improves the detection accuracy, by 5% points compared with the individual models. It achieves a weight F1 score of 0.770 on the public test set, and 0.731 on the private test set, providing a promising solution for identifying suicidal content in social media. Our analysis shows that the choice of LLMs affects the prompting performance, with larger models providing better accuracy. Our code and checkpoints are publicly available at https://github.com/khanhvynguyen/Suicide_Detection_LLMs.

自杀检测大模型应用文本分类

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。