AI专家对毁灭性风险看法分歧,根源在于安全概念认知差异。
Why do Experts Disagree on Existential Risk and P(doom)? A Survey of AI Experts
- 区分出'可控工具'与'不可控代理'两种专家观点
- 78%专家认同研究者应关注灾难性风险,仅21%知晓关键概念
- 建议先夯实安全概念基础再推进风险讨论
人工通用智能(AGI)的发展可能是人类最重要的技术突破之一。主流AI实验室和科学家呼吁全球优先关注AI安全,其潜在毁灭性风险堪比核战争。然而,关于灾难性风险与对齐问题的研究常遭专家质疑,线上争论也趋于对立。迄今尚无系统研究揭示专家信念模式与安全概念熟悉度。本调查对111位AI专家展开,考察其对安全概念的熟悉度、对安全论点的主要异议及反应。结果发现专家分为两类:'可控工具'与'不可控代理'视角,前者更重视安全。尽管78%专家认为技术研究者应关心灾难性风险,但仅21%听说过'工具趋同'(instrumental convergence),该概念指出先进AI系统会倾向追求自我保护等共性子目标。最不关注安全的群体对该概念最为陌生,表明有效沟通需从建立清晰概念基础开始。
原文摘要 · Abstract (English)
The development of artificial general intelligence (AGI) is likely to be one of humanity's most consequential technological advancements. Leading AI labs and scientists have called for the global prioritization of AI safety citing existential risks comparable to nuclear war. However, research on catastrophic risks and AI alignment is often met with skepticism, even by experts. Furthermore, online debate over the existential risk of AI has begun to turn tribal (e.g. name-calling such as "doomer" or "accelerationist"). Until now, no systematic study has explored the patterns of belief and the levels of familiarity with AI safety concepts among experts. I surveyed 111 AI experts on their familiarity with AI safety concepts, key objections to AI safety, and reactions to safety arguments. My findings reveal that AI experts cluster into two viewpoints -- an "AI as controllable tool" and an "AI as uncontrollable agent" perspective -- diverging in beliefs toward the importance of AI safety. While most experts (78%) agreed or strongly agreed that "technical AI researchers should be concerned about catastrophic risks", many were unfamiliar with specific AI safety concepts. For example, only 21% of surveyed experts had heard of "instrumental convergence," a fundamental concept in AI safety predicting that advanced AI systems will tend to pursue common sub-goals (such as self-preservation). The least concerned participants were the least familiar with concepts like this, suggesting that effective communication of AI safety should begin with establishing clear conceptual foundations in the field.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。