T-UCT让强化学习在限成本下更安全高效地决策
Threshold UCT: Cost-Constrained Monte Carlo Tree Search with Pareto Curves
- 用帕累托曲线显式建模收益与成本权衡，动态调整策略
- 在多个测试环境中，安全达标率和收益均显著优于现有方法
- 适合需严格控制成本的复杂决策场景，如自动驾驶、医疗系统
受限马尔可夫决策过程（CMDP）是应对随机不确定性下安全序列决策的主流框架，要求智能体在最大化期望收益的同时，将期望成本控制在给定阈值以下。基于蒙特卡洛树搜索（MCTS）的方法因其高效性和可扩展性，在此类问题中尤为重要。然而，现有MCTS方法要么难以找到满足约束的安全策略，要么过于保守而错失高价值策略。本文提出阈值UCT（T-UCT），一种用于CMDP规划的在线MCTS算法。与以往方法不同，T-UCT在搜索树中显式估计成本-效用权衡的帕累托曲线，并结合新的动作选择与阈值更新规则，以寻找既安全又高价值的策略。实验表明，该方法显著优于文献中的先进方法。
原文摘要 · Abstract (English)
Constrained Markov decision processes (CMDPs), in which the agent optimizes expected payoffs while keeping the expected cost below a given threshold, are the leading framework for safe sequential decision making under stochastic uncertainty. Among algorithms for planning and learning in CMDPs, methods based on Monte Carlo tree search (MCTS) have particular importance due to their efficiency and extendibility to more complex frameworks (such as partially observable settings and games). However, current MCTS-based methods for CMDPs either struggle with finding safe (i.e., constraint-satisfying) policies, or are too conservative and do not find valuable policies. We introduce Threshold UCT (T-UCT), an online MCTS-based algorithm for CMDP planning. Unlike previous MCTS-based CMDP planners, T-UCT explicitly estimates Pareto curves of cost-utility trade-offs throughout the search tree, using these together with a novel action selection and threshold update rules to seek safe and valuable policies. Our experiments demonstrate that our approach significantly outperforms state-of-the-art methods from the literature.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。