让强化学习在连续时间系统中更高效，用不确定性引导探索。
Sample-efficient and Scalable Exploration in Continuous-Time RL
- 用高斯过程等概率模型建模连续动力学，结合奖励与不确定性加权探索。
- 在奖励驱动下实现亚线性累积损失，在无奖励时给出样本复杂度上界。
- 适合需要低样本、高扩展性的连续控制任务，如机器人运动规划。
强化学习算法通常针对离散时间动态设计，尽管现实世界中的控制系统往往是连续时间的。本文研究连续时间强化学习问题，其中未知系统动力学由非线性常微分方程（ODE）表示。我们利用高斯过程和贝叶斯神经网络等概率模型，学习对底层ODE的不确定感知模型。所提出的算法COMBRL贪婪地最大化外生奖励与模型认知不确定性之加权和，从而实现可扩展且样本高效的连续时间模型基强化学习。我们证明COMBRL在奖励驱动设置下达到亚线性后悔率，在无外生奖励的无监督强化学习设置下提供了样本复杂度上界。实验表明，COMBRL在标准和无监督强化学习设置中均表现更优，相比先前方法具有更好的可扩展性和更高的样本效率，并在多个深度强化学习任务中超越基线方法。
原文摘要 · Abstract (English)
Reinforcement learning algorithms are typically designed for discrete-time dynamics, even though the underlying real-world control systems are often continuous in time. In this paper, we study the problem of continuous-time reinforcement learning, where the unknown system dynamics are represented using nonlinear ordinary differential equations (ODEs). We leverage probabilistic models, such as Gaussian processes and Bayesian neural networks, to learn an uncertainty-aware model of the underlying ODE. Our algorithm, COMBRL, greedily maximizes a weighted sum of the extrinsic reward and model epistemic uncertainty. This yields a scalable and sample-efficient approach to continuous-time model-based RL. We show that COMBRL achieves sublinear regret in the reward-driven setting, and in the unsupervised RL setting (i.e., without extrinsic rewards), we provide a sample complexity bound. In our experiments, we evaluate COMBRL in both standard and unsupervised RL settings and demonstrate that it scales better, is more sample-efficient than prior methods, and outperforms baselines across several deep RL tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。