提出无海森矩阵的双层强化学习算法，样本效率达当前最优。
Hypergradient-based Bilevel Reinforcement Learning with Improved Sample Complexity
- 基于玻尔兹曼策略优化熵正则目标，避免海森矩阵计算。
- 迭代复杂度 $O(ε^{-1})$，样本复杂度 $ ilde{O}(ε^{-2})$ 达最优。
- 无需普莱克-洛瓦日维奇条件，适用性更广，适合高效元学习场景。
双层强化学习是元学习、层次任务分解及人类反馈强化学习（RL-HF）等重要问题的形式化框架。现有算法或因使用海森矩阵导致不可扩展，或因惩罚逼近方法导致高样本复杂度。本文提出一种基于超梯度的双层强化学习算法，利用熵正则折扣回报目标下玻尔兹曼策略的最优性。该算法无需海森矩阵，满足在温和正则条件下，迭代复杂度为 $O(ε^{-1})$，样本复杂度达到 $ ilde{O}(ε^{-2})$ 的当前最优水平。此外，收敛分析中移除了先前最优工作所依赖的外层目标函数普莱克-洛瓦日维奇（PL）条件假设。
原文摘要 · Abstract (English)
Bilevel reinforcement learning (RL) is an important framework within the literature of RL that can be used to formalize various categories of problems, such as meta-learning, hierarchical task decomposition, and reinforcement learning from human feedback (RL-HF). Most of the bilevel RL algorithms are either not scalable because of using hypergradient with Hessian, or they suffer from high sample complexity because of using penalty-based approximation methods. In this work, we propose a hypergradient-based bilevel RL algorithm using the optimality of the Boltzmann policy for the entropy regularized discounted RL objective function. Our proposed algorithm is Hessian-free and obtains an iteration complexity of $O(ε^{-1})$ and state-of-the-art sample complexity of $\tilde{O}(ε^{-2})$ under mild regularity conditions. Further, in our convergence analysis, we are able to remove the assumption of the Polyak-Lojasiewicz (PL) condition on the outer-level objective function present in the prior state-of-the-art sample complexity work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。