分析智能体AI认知能力带来的风险，提出分级管控策略。
Understanding Cognition-Induced Risks in Agentic AI Systems
- 按物理、社会、自我三层次分析认知风险
- 揭示认知增强削弱人类自主权与控制力
- 适合关注AI安全与治理的研究者
由大语言模型驱动的前沿智能体系统展现出类人认知特征。随着这些系统在各领域深度集成，其认知参与引发了对人类社会的重大关切，但相关研究仍不充分。为填补这一空白，我们基于认知范围的三个层级——物理认知、社会认知和自我参照认知，系统分析了认知能力扩展所引发的风险。针对每一层级，探讨其对人类自主性、代理权和控制能力的潜在威胁，并最终提出缓解策略，以增强智能体AI系统的可控性，保障其长期安全发展。
原文摘要 · Abstract (English)
Frontier agentic systems powered by large language models (LLMs) exhibit human-like patterns of cognition. As these systems become deeply integrated across different domains, their cognitive engagement raises critical concerns for human society that remain insufficiently studied. To address this gap, we systematically analyze risks induced by expanding cognitive capabilities, following a three-level framework defined by their cognitive scope, from physical cognition to social cognition, and finally to self-referential cognition. We study their potential risks to human agency, autonomy, and control capability, corresponding to each cognitive level. We finally propose strategies to mitigate these risks and enhance the controllability of agentic AI systems, ensuring their long-term safe development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。