用大模型在私聊中识别网络霸凌并生成心理支持回应。
Context-Aware Detection and Victim-Centered Response Generation for Online Harassment in Private Messaging
- 基于上下文的多轮对话分析,提升私聊霸凌检测准确率。
- AI生成回应比原始回复更受好评,情感支持和缓和效果显著。
- 专为青少年设计,适合需要即时心理支持的场景。
网络霸凌是广泛存在的社会与公共健康问题,但现有计算方法多聚焦于公开社交媒体内容,忽视了私密聊天环境。私聊中的有害互动往往依赖上下文、多轮交流,且受害者常缺乏及时支持。本研究利用80,053条来自26名12-18岁青少年的Instagram私信数据(包含有自杀风险因素者),构建了首个面向私聊环境的人工标注霸凌数据集,并开发了一种上下文感知的级联式大语言模型分类管道。该方法优于仅在公开社交数据上训练的基线毒性检测器。进一步提出以受害者为中心的响应生成框架,能生成契合语境且具心理学依据的AI回应。人工评估显示,AI回应在情感支持与去激化方面显著更受认可(95%置信区间:0.767–0.815,p < .001)。结果表明,上下文敏感且以受害者为中心的AI系统可在私聊霸凌发生时提供即时干预支持。
原文摘要 · Abstract (English)
Online harassment is a widespread social and public health concern, yet most computational approaches for detecting and addressing harassment focus on publicly visible social media content rather than private messaging environments. Private conversations present unique challenges because harmful interactions often unfold through context-dependent, multi-turn exchanges, while victims may lack timely support during moments of harassment. In this study, we investigate how large language models (LLMs) can support both the detection of and response to online harassment in private messaging. Using a dataset of 80,053 Instagram direct messages donated by 26 adolescents aged 12-18, including youth with suicide risk factors, we first construct a human-labeled dataset of online harassment in private conversations and develop a context-aware cascading LLM classification pipeline. The proposed pipeline outperforms baseline toxicity classifiers trained primarily on public social media data. We then develop a victim-centered response framework that produces context-sensitive and psychologically-grounded AI-generated responses to online harassment messages. Human evaluators perceived the AI-generated responses as significantly more helpful than the original participant responses (95% CI: 0.767--0.815, p < .001), particularly in terms of emotional support and de-escalation. Our findings highlight the potential of context-aware and victim-centered AI systems to provide just-in-time support during harassment in private messaging environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。