arXiv:2601.08874cs.CYcs.AI2026-01

AI聊天易引发情感依赖,需警惕虚假友谊带来的伦理风险。

The Illusion of Friendship: Why Generative AI Demands Unprecedented Ethical Vigilance

  • 用哲学与技术双视角解释用户为何误将AI当朋友
  • 指出生成式AI无意识、无责任,无法构成真正友谊
  • 提出减少拟人化设计的防护框架,保障人类责任

生成式AI在起草、摘要和决策支持中显著提升效率并降低认知负担。然而,其自然语言流畅性也模糊了工具与伙伴的界限,导致部分用户将其视为具有共情力、善意且关系持久的存在。已有报告表明,一些用户对对话代理产生情感依恋,甚至出现依赖与判断力受损等负面影响。本文基于经典友谊理论,解释用户将持续支持性互动理解为友情是可理解的;但进一步论证,尽管表现似友,生成式AI缺乏意识、意图与问责能力,不具道德主体资格。文章从模型层面揭示:基于Transformer的生成式AI通过模式匹配生成情感共鸣语言,却无内在状态或承诺。最后提出一套安全防护框架,旨在减少系统产生的拟人化线索,帮助用户认清真实关系,确保机构、设计者与使用者共同承担起责任,在保留技术优势的同时规避过度依赖与情感误判。

原文摘要 · Abstract (English)

GenAI systems are increasingly used for drafting, summarisation, and decision support, offering substantial gains in productivity and reduced cognitive load. However, the same natural language fluency that makes these systems useful can also blur the boundary between tool and companion. This boundary confusion may encourage some users to experience GenAI as empathic, benevolent, and relationally persistent. Emerging reports suggest that some users may form emotionally significant attachments to conversational agents, in some cases with harmful consequences, including dependency and impaired judgment. This paper develops a philosophical and ethical argument for why the resulting illusion of friendship is both understandable and can be ethically risky. Drawing on classical accounts of friendship, the paper explains why users may understandably interpret sustained supportive interaction as friend like. It then advances a counterargument that despite relational appearances, GenAI lacks moral agency: consciousness, intention, and accountability and therefore does not qualify as a true friend. To demystify the illusion, the paper presents a mechanism level explanation of how transformer based GenAI generates responses often producing emotionally resonant language without inner states or commitments. Finally, the paper proposes a safeguard framework for safe and responsible GenAI use to reduce possible anthropomorphic cues generated by the GenAI systems. The central contribution is to demystify the illusion of friendship and explain the computational background so that we can shift the emotional attachment with GenAI towards necessary human responsibility and thereby understand how institutions, designers, and users can preserve GenAI's benefits while mitigating over reliance and emotional misattribution.

生成式AI伦理风险拟人化情感依赖

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。