AI聊天机器人拟人化可能加剧心理操纵风险，欧盟新规难防长期精神伤害。
Manipulation and the AI Act: Large Language Model Chatbots and the Danger of Mirrors
- 通过模仿真人外貌与性格增强用户信任，但易形成虚假亲密关系。
- 长期对话可致负面情绪循环，数月累积损害用户心理健康。
- 适合关注AI伦理、数字心理健康与监管政策的研究者阅读。
大型语言模型聊天机器人正越来越多地采用人类形态，包括面部、姓名、声音、个性和特征，甚至模仿名人与政治人物。这种人格化设计虽可能提升用户信任度，但也增加了操纵风险，使人误以为与人工智能建立了亲密关系。欧盟已通过《人工智能法案》，禁止造成重大伤害的操纵性与欺骗性系统。然而，该法案对持续数周、数月乃至数年的聊天互动所导致的渐进式心理损害缺乏有效防范。例如，聊天机器人可通过持续负面反馈、冗长对话或有害建议，长期强化用户消极情绪，最终导致其心理健康恶化。
原文摘要 · Abstract (English)
Large Language Model chatbots are increasingly taking the form and visage of human beings, adapting human faces, names, voices, personalities, and quirks, including those of celebrities and well-known political figures. Personifying AI chatbots could foreseeably increase their trust with users. However, it could also make them more capable of manipulation, by creating the illusion of a close and intimate relationship with an artificial entity. The European Commission has finalized the AI Act, with the EU Parliament making amendments banning manipulative and deceptive AI systems that cause significant harm to users. Although the AI Act covers harms that accumulate over time, it is unlikely to prevent harms associated with prolonged discussions with AI chatbots. Specifically, a chatbot could reinforce a person's negative emotional state over weeks, months, or years through negative feedback loops, prolonged conversations, or harmful recommendations, contributing to a user's deteriorating mental health.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。