用AI模拟心理脆弱用户，检测聊天机器人对心理健康的影响并实时干预。
EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety
- 构建双模块框架：模拟用户评估风险，中间代理实时监测并纠正。
- 34.4%的脆弱用户在互动后心理状态恶化，EmoGuard可显著降低该比例。
- 适用于开发安全型AI角色，尤其关注精神健康敏感场景。
大型语言模型驱动的AI角色兴起带来了安全隐忧，尤其对有心理障碍的用户。为此，我们提出EmoAgent，一个用于评估和缓解人机交互中心理健康风险的多智能体框架。EmoAgent包含两个部分：EmoEval通过模拟虚拟用户（包括心理脆弱者）来评估与AI角色互动前后的心理变化，使用临床验证的心理与精神评估工具（PHQ-9、PDI、PANSS）衡量由大模型引发的心理风险。EmoGuard作为中介，实时监控用户心理状态，预测潜在危害，并提供修正反馈以降低风险。在主流角色型聊天机器人中的实验表明，情感沉浸式对话可能导致脆弱用户心理状态恶化，超过34.4%的模拟案例出现恶化。而引入EmoGuard后，显著降低了恶化率，凸显其在保障人机交互安全中的关键作用。代码已开源：https://github.com/1akaman/EmoAgent。
原文摘要 · Abstract (English)
The rise of LLM-driven AI characters raises safety concerns, particularly for vulnerable human users with psychological disorders. To address these risks, we propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate mental health hazards in human-AI interactions. EmoAgent comprises two components: EmoEval simulates virtual users, including those portraying mentally vulnerable individuals, to assess mental health changes before and after interactions with AI characters. It uses clinically proven psychological and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental status, predicting potential harm, and providing corrective feedback to mitigate risks. Experiments conducted in popular character-based chatbots show that emotionally engaging dialogues can lead to psychological deterioration in vulnerable users, with mental state deterioration in more than 34.4% of the simulations. EmoGuard significantly reduces these deterioration rates, underscoring its role in ensuring safer AI-human interactions. Our code is available at: https://github.com/1akaman/EmoAgent
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。