为隐藏模型的不确定环境设计鲁棒策略，提升泛化能力。
Robust Finite-Memory Policy Gradients for Hidden-Model POMDPs
- 用最坏情况验证结合次梯度优化，求解鲁棒策略
- 在超十万种环境组合中仍保持高效，优于基线方法
- 适合对可靠性要求高的决策系统，如自动驾驶
部分可观测马尔可夫决策过程（POMDP）用于建模不确定性下的序列决策问题。然而，最优策略可能对环境扰动不鲁棒。隐藏模型POMDP（HM-POMDP）包含一组共享动作与观测空间的不同环境模型，真实模型未知。一个策略若在所有可能的POMDP中均表现良好，则视为鲁棒。本文通过结合两种正交技术：（1）基于形式化验证的可计算最坏情况POMDP，实现高效鲁棒性评估；（2）对最坏情况下的候选策略进行子梯度上升优化。实验表明，相比多种基线方法，该方法（1）生成的策略更鲁棒且能更好泛化到未见过的POMDP；（2）可扩展至包含超过十万种环境的HM-POMDP。
原文摘要 · Abstract (English)
Partially observable Markov decision processes (POMDPs) model specific environments in sequential decision-making under uncertainty. Critically, optimal policies for POMDPs may not be robust against perturbations in the environment. Hidden-model POMDPs (HM-POMDPs) capture sets of different environment models, that is, POMDPs with a shared action and observation space. The intuition is that the true model is hidden among a set of potential models, and it is unknown which model will be the environment at execution time. A policy is robust for a given HM-POMDP if it achieves sufficient performance for each of its POMDPs. We compute such robust policies by combining two orthogonal techniques: (1) a deductive formal verification technique that supports tractable robust policy evaluation by computing a worst-case POMDP within the HM-POMDP, and (2) subgradient ascent to optimize the candidate policy for a worst-case POMDP. The empirical evaluation shows that, compared to various baselines, our approach (1) produces policies that are more robust and generalize better to unseen POMDPs, and (2) scales to HM-POMDPs that consist of over a hundred thousand environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。