让强化学习在未知环境中更安全，通过动态估计阈值来控制风险。
Ensuring Safety in an Uncertain Environment: Constrained MDPs via Stochastic Thresholds
- 用滚动窗口估计环境中的随机阈值，实现对不确定性的建模。
- 理论证明算法在T轮中奖励损失和约束违规均不超过√T阶增长。
- 首次在阈值未知时保证性能，适合高安全性需求的智能系统。
本文研究具有随机阈值约束的约束马尔可夫决策过程（CMDPs），旨在提升强化学习在未知与不确定环境中的安全性。我们利用从与不确定环境交互中采样的滚动窗口估计器来估计阈值，并基于此设计了新的基于模型的原始-对偶算法——随机悲观-乐观阈值化（SPOT），以应对多个针对随机阈值的约束。SPOT支持在悲观与乐观阈值设定下进行强化学习。理论证明该算法实现次线性遗憾与约束违规：在T轮中，奖励遗憾为$ ilde{ ext{O}}( ext{√}T)$，同时允许$ ilde{ ext{O}}( ext{√}T)$阶的约束违规。理论结果表明，其性能可媲美依赖固定明确阈值的方法。据我们所知，SPOT是首个在阈值本身未知的不确定性环境中实现理论性能保证的强化学习算法。
原文摘要 · Abstract (English)
This paper studies constrained Markov decision processes (CMDPs) with constraints against stochastic thresholds, aiming at safety of reinforcement learning in unknown and uncertain environments. We leverage a Growing-Window estimator sampling from interactions with the uncertain environment to estimate the thresholds, based on which we design Stochastic Pessimistic-Optimistic Thresholding (SPOT), a novel model-based primal-dual algorithm for multiple constraints against stochastic thresholds. SPOT enables reinforcement learning under both pessimistic and optimistic threshold settings. We prove that our algorithm achieves sublinear regret and constraint violation; i.e., a reward regret of $\tilde{\mathcal{O}}(\sqrt{T})$ while allowing an $\tilde{\mathcal{O}}(\sqrt{T})$ constraint violation over $T$ episodes. The theoretical guarantees show that our algorithm achieves performance comparable to that of an approach relying on fixed and clear thresholds. To the best of our knowledge, SPOT is the first reinforcement learning algorithm that realises theoretical guaranteed performance in an uncertain environment where even thresholds are unknown.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。