用深度强化学习训练可验证的有限状态控制器，解决大规模部分可观测决策问题。
Finite-State Controllers for (Hidden-Model) POMDPs using Deep Reinforcement Learning
- 用RNN神经策略+高效提取，生成可形式化验证的有限状态控制器
- 在大规模状态空间上优于当前最优的POMDP求解器
- 适用于多模型鲁棒决策场景，适合需要安全保证的应用
求解部分可观测马尔可夫决策过程（POMDP）需在不完整状态信息下计算策略。尽管近期取得进展，现有POMDP求解器的可扩展性仍受限。许多场景要求策略对多个POMDP具有鲁棒性，进一步加剧可扩展性挑战。本文提出Lexpop框架用于求解POMDP。Lexpop（1）利用深度强化学习训练由循环神经网络表示的神经策略，（2）通过高效提取方法构建模仿神经策略的有限状态控制器。关键在于，与神经策略不同，此类控制器可被形式化评估，提供性能保证。我们还将Lexpop扩展至求解隐模型POMDP（HM-POMDP），描述有限个POMDP的集合。为每个提取的控制器关联其最差情况下的POMDP，利用一组此类POMDP，迭代训练鲁棒神经策略并提取鲁棒控制器。实验表明，在具有大状态空间的问题上，Lexpop优于当前最先进的POMDP及HM-POMDP求解器。
原文摘要 · Abstract (English)
Solving partially observable Markov decision processes (POMDPs) requires computing policies under imperfect state information. Despite recent advances, the scalability of existing POMDP solvers remains limited. Moreover, many settings require a policy that is robust across multiple POMDPs, further aggravating the scalability issue. We propose the Lexpop framework for POMDP solving. Lexpop (1) employs deep reinforcement learning to train a neural policy, represented by a recurrent neural network, and (2) constructs a finite-state controller mimicking the neural policy through efficient extraction methods. Crucially, unlike neural policies, such controllers can be formally evaluated, providing performance guarantees. We extend Lexpop to compute robust policies for hidden-model POMDPs (HM-POMDPs), which describe finite sets of POMDPs. We associate every extracted controller with its worst-case POMDP. Using a set of such POMDPs, we iteratively train a robust neural policy and consequently extract a robust controller. Our experiments show that on problems with large state spaces, Lexpop outperforms state-of-the-art solvers for POMDPs as well as HM-POMDPs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。