AI代理风险应按自主程度监管，而非仅看计算规模。
AI Agents Should be Regulated Based on the Extent of Their Autonomous Operations
- 以行动序列衡量自主性，更准确反映潜在影响。
- 长期规划的智能体可能引发人类灭绝级风险。
- 适合关注AI安全与政策制定的研究者阅读。
本文主张，应根据AI代理的自主运行程度进行监管。具备长期规划和战略能力的AI代理可能带来人类灭绝和不可逆全球灾难的重大风险。现有监管多以计算规模作为危害潜力的代理指标，但对此类主要依赖推理时计算能力的代理而言，该方法不足。为此，我们讨论了科学家关于生存性风险的相关法规与建议，并指出行动序列（反映代理自主程度）比依赖观测环境状态的现有度量方式，更能适切评估潜在影响。
原文摘要 · Abstract (English)
This position paper argues that AI agents should be regulated by the extent to which they operate autonomously. AI agents with long-term planning and strategic capabilities can pose significant risks of human extinction and irreversible global catastrophes. While existing regulations often focus on computational scale as a proxy for potential harm, we argue that such measures are insufficient for assessing the risks posed by agents whose capabilities arise primarily from inference-time computation. To support our position, we discuss relevant regulations and recommendations from scientists regarding existential risks, as well as the advantages of using action sequences -- which reflect the degree of an agent's autonomy -- as a more suitable measure of potential impact than existing metrics that rely on observing environmental states.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。