用自适应探索策略让AI在动态任务中快速学习并持续进步
Lipschitz Lifelong Monte Carlo Tree Search for Mastering Non-Stationary Tasks
- 基于任务间平滑变化特性，动态调整搜索策略以复用经验
- 实验显示收敛速度比现有方法快3至4倍，奖励提升显著
- 适合需要长期适应环境变化的智能体系统，如机器人决策
蒙特卡洛树搜索（MCTS）通过上限置信区间树（UCT）在探索与利用之间取得平衡，已在复杂规划任务中表现优异。然而，现有研究未考虑基于MCTS的终身规划问题，即智能体在运行生命周期中连续面对非平稳任务——例如转移概率和奖励随时间变化。本文提出LiZero，一种基于利普希茨连续性的终身规划方法。我们引入自适应UCT（aUCT），根据任务间的利普希茨连续性以及蒙特卡洛动作采样中的置信度，将源任务的知识迁移至新任务的探索与利用过程。我们分析了LiZero在采样效率上的加速因子，并提出了数据驱动与模型驱动相结合的在线算法，以高效计算aUCT，同时给出了采样复杂度与误差界。实验结果表明，与现有MCTS及终身学习基线相比，LiZero在达到最优奖励时收敛速度提升了3~4倍。结果凸显了LiZero在动态现实环境中推进决策与规划的潜力。
原文摘要 · Abstract (English)
Monte Carlo Tree Search (MCTS) has proven highly effective in solving complex planning tasks by balancing exploration and exploitation using Upper Confidence Bound for Trees (UCT). However, existing work have not considered MCTS-based lifelong planning, where an agent faces a non-stationary series of tasks -- e.g., with varying transition probabilities and rewards -- that are drawn sequentially throughout the operational lifetime. This paper presents LiZero for Lipschitz lifelong planning using MCTS. We propose a novel concept of adaptive UCT (aUCT) to transfer knowledge from a source task to the exploration/exploitation of a new task, depending on both the Lipschitz continuity between tasks and the confidence of knowledge in in Monte Carlo action sampling. We analyze LiZero's acceleration factor in terms of improved sampling efficiency and also develop efficient algorithms to compute aUCT in an online fashion by both data-driven and model-based approaches, whose sampling complexity and error bounds are also characterized. Experiment results show that LiZero significantly outperforms existing MCTS and lifelong learning baselines in terms of much faster convergence (3$\sim$4x) to optimal rewards. Our results highlight the potential of LiZero to advance decision-making and planning in dynamic real-world environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。