用新方法让AI在不断变化的环境中持续学习控制策略。
Optimal control of the future via prospective learning with control
- 将监督学习扩展到非平稳、无重置环境中的控制学习
- 理论上证明该方法可渐近逼近贝叶斯最优策略
- 在动态觅食任务中，比传统强化学习快数十倍
未来最优控制是人工智能的前沿挑战。现有方法多基于强化学习（RL），但其依赖静态环境与周期性重置，限制了实际应用。本文提出“前瞻性控制学习”（Prospective Learning with Control, PLuC）框架，将监督学习拓展至非平稳、无重置环境中的控制学习。在若干合理假设下，我们证明经验风险最小化（ERM）可渐近达到贝叶斯最优策略。进一步以觅食任务为例，该框架作为典型应用场景，表明传统强化学习算法因假设静态环境而表现不佳；即使加入时间感知改进，其收敛速度仍比我们的前瞻性觅食代理慢一个数量级以上。代码已开源：https://github.com/neurodata/procontrol。
原文摘要 · Abstract (English)
Optimal control of the future is the next frontier for AI. Current approaches to this problem are typically rooted in reinforcement learning (RL). RL is mathematically distinct from supervised learning, which has been the main workhorse for the recent achievements in AI. Moreover, RL typically operates in a stationary environment with episodic resets, limiting its utility. Here, we extend supervised learning to address learning to control in non-stationary, reset-free environments. Using this framework, called ''Prospective Learning with Control'' (PLuC), we prove that under certain fairly general assumptions, empirical risk minimization (ERM) asymptotically achieves the Bayes optimal policy. We then consider a specific instance of prospective learning with control: foraging, a canonical task relevant to both natural and artificial agents. We illustrate that modern RL algorithms, which assume stationarity, struggle in these non-stationary reset-free environments. Even with time-aware modifications, they converge orders of magnitude slower than our prospective foraging agents on a simple 1-D foraging benchmark. Code is available at: https://github.com/neurodata/procontrol.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。