提出在线鲁棒规划算法RSS,应对模型不确定带来的安全问题。
Online Robust Planning under Model Uncertainty: A Sample-Based Approach
- 基于样本平均近似构造鲁棒价值函数,提升不确定性下的决策稳定性
- 理论保证在有限样本下性能,且计算复杂度与状态空间无关
- 适用于连续状态空间,适合动态环境中实时规划
在马尔可夫决策过程(MDP)中,在线规划通过从当前状态模拟未来轨迹来做出序列决策,适用于大规模或动态环境。基于样本的方法如稀疏采样和蒙特卡洛树搜索(MCTS)利用生成模型近似最优动作,但在实际中生成模型常由有限数据学习而来,引入近似误差,导致性能下降或不安全行为。为应对这一问题,鲁棒MDP(RMDP)提供了一个处理模型不确定性的规范框架,但现有方法通常计算开销大,难以实时使用。本文提出首个具有有限样本理论性能保证的在线RMDP规划算法——鲁棒稀疏采样(RSS)。不同于仅估计名义价值函数的稀疏采样,RSS利用样本平均近似(SAA)的高效性与理论性质,计算鲁棒价值函数,实现在线环境下可计算的鲁棒策略。该方法适用于无限或连续状态空间,其样本与计算复杂度均独立于状态空间大小。我们提供了理论性能保证,并实证表明在动态不确定性环境下,RSS优于标准稀疏采样。
原文摘要 · Abstract (English)
Online planning in Markov Decision Processes (MDPs) enables agents to make sequential decisions by simulating future trajectories from the current state, making it well-suited for large-scale or dynamic environments. Sample-based methods such as Sparse Sampling and Monte Carlo Tree Search (MCTS) are widely adopted for their ability to approximate optimal actions using a generative model. However, in practical settings, the generative model is often learned from limited data, introducing approximation errors that can degrade performance or lead to unsafe behaviors. To address these challenges, Robust MDPs (RMDPs) offer a principled framework for planning under model uncertainty, yet existing approaches are typically computationally intensive and not suited for real-time use. In this work, we introduce Robust Sparse Sampling (RSS), the first online planning algorithm for RMDPs with finite-sample theoretical performance guarantees. Unlike Sparse Sampling, which estimates the nominal value function, RSS computes a robust value function by leveraging the efficiency and theoretical properties of Sample Average Approximation (SAA), enabling tractable robust policy computation in online settings. RSS is applicable to infinite or continuous state spaces, and its sample and computational complexities are independent of the state space size. We provide theoretical performance guarantees and empirically show that RSS outperforms standard Sparse Sampling in environments with uncertain dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。