提出自适应算法ZoRL，显著降低连续空间强化学习的累计损失。
Provably Adaptive Average Reward Reinforcement Learning for Metric Spaces
- 基于状态-动作空间动态聚焦，自适应缩小探索范围
- 理论证明后悔上界为T^{1 - d_eff^{-1}}，优于固定划分方法
- 适合状态空间复杂但结构良好的强化学习任务
我们研究了利普希茨马尔可夫决策过程（Lipschitz MDPs）下的无限时域平均奖励强化学习，该类问题包含线性、RKHS MDPs及函数逼近框架等重要情形。提出自适应算法ZoRL，其后悔上界为\mathcal{O}(T^{1 - d_{\text{eff.}}^{-1}})，其中d_{\text{eff.}} = 2d_\mathcal{S} + d_z + 3，d_\mathcal{S}为状态空间维数，d_z为缩放维数。相比固定离散化算法（此时d_{\text{eff.}} = 2(d_\mathcal{S} + d_\mathcal{A}) + 2，d_\mathcal{A}为动作空间维数），ZoRL通过自适应离散化并聚焦于“有前景区域”实现更优性能。缩放维数d_z是依赖问题的量，上限为状态-动作空间维度，若MDP具有良性结构，则ZoRL的后悔将较小。实验表明ZoRL优于现有先进算法，验证了自适应带来的收益。
原文摘要 · Abstract (English)
We study infinite-horizon average-reward reinforcement learning (RL) for Lipschitz MDPs, a broad class that subsumes several important classes such as linear and RKHS MDPs, function approximation frameworks, and develop an adaptive algorithm $\text{ZoRL}$ with regret bounded as $\mathcal{O}\big(T^{1 - d_{\text{eff.}}^{-1}}\big)$, where $d_{\text{eff.}}= 2d_\mathcal{S} + d_z + 3$, $d_\mathcal{S}$ is the dimension of the state space and $d_z$ is the zooming dimension. In contrast, algorithms with fixed discretization yield $d_{\text{eff.}} = 2(d_\mathcal{S} + d_\mathcal{A}) + 2$, $d_\mathcal{A}$ being the dimension of action space. $\text{ZoRL}$ achieves this by discretizing the state-action space adaptively and zooming into ''promising regions'' of the state-action space. $d_z$, a problem-dependent quantity bounded by the state-action space's dimension, allows us to conclude that if an MDP is benign, then the regret of $\text{ZoRL}$ will be small. The zooming dimension and $\text{ZoRL}$ are truly adaptive, i.e., the current work shows how to capture adaptivity gains for infinite-horizon average-reward RL. $\text{ZoRL}$ outperforms other state-of-the-art algorithms in experiments, thereby demonstrating the gains arising due to adaptivity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。