将部分可观测问题转为有限状态马尔可夫决策过程，用强化学习高效求解。
Scalable Policy-Based RL Algorithms for POMDPs
- 将信念状态离散化为超状态，构建等价的有限状态MDP。
- 历史长度越长，近似误差呈指数下降，理论保证更强。
- 适合处理复杂观测环境下的决策问题，如机器人控制。
POMDP中信念状态的连续性带来了巨大的计算挑战。本文提出将部分可观测强化学习（PORL）问题通过将原POMDP模型近似为有限状态马尔可夫决策过程（称为超状态MDP）来求解。我们推导了理论保证，改进了先前工作，建立了超状态MDP最优价值函数与原始POMDP之间的关系。接着，采用基于策略的学习方法结合线性函数逼近，在超状态MDP上学习最优策略。结果表明，可通过时序差分学习（TD-learning）与策略优化联合求解该问题，其中状态对应于有限历史。我们证明，近似误差随历史长度呈指数衰减。据我们所知，这是首个显式量化在非马尔可夫动态下应用标准TD学习引入误差的有限时间界。
原文摘要 · Abstract (English)
The continuous nature of belief states in POMDPs presents significant computational challenges in learning the optimal policy. In this paper, we consider an approach that solves a Partially Observable Reinforcement Learning (PORL) problem by approximating the corresponding POMDP model into a finite-state Markov Decision Process (MDP) (called Superstate MDP). We first derive theoretical guarantees that improve upon prior work that relate the optimal value function of the transformed Superstate MDP to the optimal value function of the original POMDP. Next, we propose a policy-based learning approach with linear function approximation to learn the optimal policy for the Superstate MDP. Consequently, our approach shows that a POMDP can be approximately solved using TD-learning followed by Policy Optimization by treating it as an MDP, where the MDP state corresponds to a finite history. We show that the approximation error decreases exponentially with the length of this history. To the best of our knowledge, our finite-time bounds are the first to explicitly quantify the error introduced when applying standard TD learning to a setting where the true dynamics are not Markovian.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。