提出自适应策略组合，在不确定环境下动态选择最优策略。
Adaptive Policy Portfolios for Robust Markov Decision Processes
- 离线构建多策略集合，线上轻量级选择最佳策略
- 在不可知环境下的策略损失比最优策略最多差10%（理论保证）
- 适合需要高鲁棒性的机器人控制与决策系统
鲁棒马尔可夫决策过程通过优化单一策略来应对一组可能的状态转移函数。当未知动态在部署后部分可识别时，这种做法可能过于保守。本文研究自适应策略组合：离线合成的有限记忆无状态随机策略集，搭配轻量在线选择器。鲁棒遗憾是衡量组合质量的自然指标——对每个可能环境，它衡量最优组合成员相对于已知该环境时应选最优策略的损失。相关遗憾目标曾由Ghavamzadeh等人（2016）研究，侧重于安全策略改进的近似与松弛。本文给出组合认证与合成的复杂性理论分析：认证一个给定组合在无环（s,a）矩形RMDPs中已是∀ℝ-完全；合成大小为一元有界的组合在一般有理多面体下为∃∀ℝ-完全，即使折扣因子固定且动态无环。单策略情形本身在组合与代数上均困难。最后，提出一种便于运行时特化的离线组合构造方法。
原文摘要 · Abstract (English)
Robust Markov decision processes optimize one policy against a set of plausible transition functions. This can be conservative when the unknown dynamics are fixed and become partially identifiable after deployment. We study adaptive policy portfolios: finite sets of memoryless randomized policies synthesized offline and paired with a lightweight online selector. Robust regret is a natural measure of portfolio quality: for each plausible environment, it measures the loss of the best portfolio member relative to the policy that would have been optimal had that environment been known. Related regret objectives were studied by Ghavamzadeh et al. (2016) with an emphasis on approximations and relaxations for safe policy improvement. We give a complexity-theoretic account of portfolio certification and synthesis. Certifying a given portfolio is $\forall\mathbb{R}$-complete already for deterministic portfolios in acyclic (s,a)-rectangular RMDPs. Synthesizing a portfolio of unary-bounded size is $\exists\forall\mathbb{R}$-complete for general rational polytopes, even with fixed discount and acyclic dynamics. The single-policy case is already hard, both combinatorially and algebraically. Finally, we present an offline portfolio construction that is amenable to runtime specialization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。