arXiv:2501.13075cs.AI2025-01被引 12

机器学习忽视了应对未知未来的鲁棒性,这可能是其在开放世界中脆弱的根源。

Evolution and The Knightian Blindspot of Machine Learning

  • 用生物进化对比强化学习,揭示其对未知不确定性的忽视
  • 进化能适应极端分布外情况,而当前强化学习难以实现零样本迁移
  • 提出应将不可量化不确定性纳入算法设计,适合追求鲁棒AI的研究者

本文指出,机器学习(ML)普遍忽略了通用智能的关键特征:在开放世界中对定性未知未来的鲁棒性。这种鲁棒性对应于经济学中的奈特不确定性(Knightian uncertainty, KU),即无法量化的不确定性,而这类不确定性被排除在机器学习的核心形式化框架之外。本文旨在识别这一盲点,论证其重要性,并推动研究以解决该问题,我们认为这是实现真正鲁棒的开放世界人工智能所必需的。为阐明这一盲点,我们对比了强化学习(RL)与生物进化过程。尽管进展显著,但RL在开放世界中仍表现不佳,常因未预见情境而失败。例如,将仅在美国训练的自动驾驶策略直接零样本迁移到英国,目前看来极为不切实际。与此形成鲜明对比的是,生物进化能够持续产生在开放世界中繁荣的主体,甚至适应显著分布外的情况(如入侵物种;或人类,其可实现零样本国际驾驶)。有趣的是,进化通过非显式理论、形式化或数学梯度实现了此类鲁棒性。我们分析了强化学习典型形式化的假设,揭示其如何限制其应对复杂世界中未知未知的能力。此外,我们识别出进化过程促进对新奇和不可预测挑战鲁棒性的机制,并探讨将其算法化实现的潜在路径。结论是,机器学习令人遗憾的剩余脆弱性可能源于其形式化中的盲点,直接面对奈特不确定性挑战或将带来显著提升。

原文摘要 · Abstract (English)

This paper claims that machine learning (ML) largely overlooks an important facet of general intelligence: robustness to a qualitatively unknown future in an open world. Such robustness relates to Knightian uncertainty (KU) in economics, i.e. uncertainty that cannot be quantified, which is excluded from consideration in ML's key formalisms. This paper aims to identify this blind spot, argue its importance, and catalyze research into addressing it, which we believe is necessary to create truly robust open-world AI. To help illuminate the blind spot, we contrast one area of ML, reinforcement learning (RL), with the process of biological evolution. Despite staggering ongoing progress, RL still struggles in open-world situations, often failing under unforeseen situations. For example, the idea of zero-shot transferring a self-driving car policy trained only in the US to the UK currently seems exceedingly ambitious. In dramatic contrast, biological evolution routinely produces agents that thrive within an open world, sometimes even to situations that are remarkably out-of-distribution (e.g. invasive species; or humans, who do undertake such zero-shot international driving). Interestingly, evolution achieves such robustness without explicit theory, formalisms, or mathematical gradients. We explore the assumptions underlying RL's typical formalisms, showing how they limit RL's engagement with the unknown unknowns characteristic of an ever-changing complex world. Further, we identify mechanisms through which evolutionary processes foster robustness to novel and unpredictable challenges, and discuss potential pathways to algorithmically embody them. The conclusion is that the intriguing remaining fragility of ML may result from blind spots in its formalisms, and that significant gains may result from direct confrontation with the challenge of KU.

机器学习鲁棒性进化开放世界

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。