arXiv:2409.20521cs.LGcs.AI2024-09被引 15

提出新算法提升鲁棒强化学习效率,理论最优且计算更省。

Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learning

  • 设计抗环境扰动的分布鲁棒算法,基于线性MDP建模动态不确定性。
  • 理论误差比现有方法降低约 $\mathcal{O}(dH/\min\{1/ρ,H\})$,近似最优。
  • 支持罕见策略切换,大幅减少计算调用次数,适合高效部署场景。

我们研究离线动态强化学习(Off-dynamics RL),即策略训练与部署环境不同。为应对环境扰动,基于分布鲁棒马尔可夫决策过程(DRMDPs)框架,假设名义与扰动动态均为线性马尔可夫决策过程。提出新算法 We-DRIVE-U,其平均次优性为 $\widetilde{\mathcal{O}}\big({d H \cdot \min \{1/ρ, H\}/\sqrt{K} }\big)$,其中 $K$ 为训练轮数,$H$ 为规划长度,$d$ 为特征维度,$ρ$ 为不确定性水平。该结果较现有最优方法改进 $\mathcal{O}(dH/\min\{1/ρ,H\})$。同时构造首个信息论下界,表明算法在任意 $ρ\in(0,1]$ 下近似最优,仅差 $\mathcal{O}(\sqrt{H})$ 因子。算法采用‘稀疏切换’设计,仅需 $\mathcal{O}(dH\log(1+H^2K))$ 次策略切换及 $\mathcal{O}(d^2H\log(1+H^2K))$ 次对偶优化求解,显著优于现有算法中 $\mathcal{O}(K)$ 的复杂度。

原文摘要 · Abstract (English)

We study off-dynamics Reinforcement Learning (RL), where the policy training and deployment environments are different. To deal with this environmental perturbation, we focus on learning policies robust to uncertainties in transition dynamics under the framework of distributionally robust Markov decision processes (DRMDPs), where the nominal and perturbed dynamics are linear Markov Decision Processes. We propose a novel algorithm We-DRIVE-U that enjoys an average suboptimality $\widetilde{\mathcal{O}}\big({d H \cdot \min \{1/ρ, H\}/\sqrt{K} }\big)$, where $K$ is the number of episodes, $H$ is the horizon length, $d$ is the feature dimension and $ρ$ is the uncertainty level. This result improves the state-of-the-art by $\mathcal{O}(dH/\min\{1/ρ,H\})$. We also construct a novel hard instance and derive the first information-theoretic lower bound in this setting, which indicates our algorithm is near-optimal up to $\mathcal{O}(\sqrt{H})$ for any uncertainty level $ρ\in(0,1]$. Our algorithm also enjoys a 'rare-switching' design, and thus only requires $\mathcal{O}(dH\log(1+H^2K))$ policy switches and $\mathcal{O}(d^2H\log(1+H^2K))$ calls for oracle to solve dual optimization problems, which significantly improves the computational efficiency of existing algorithms for DRMDPs, whose policy switch and oracle complexities are both $\mathcal{O}(K)$.

强化学习鲁棒性优化算法MDP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。