融合强化学习与动作模型学习，提升复杂任务的长期规划能力。
Integrating Reinforcement Learning, Action Model Learning, and Numeric Planning for Tackling Complex Tasks
- 用NSAM算法学习数值领域模型，结合规划器实现模型驱动决策。
- 在Minecraft任务中，该方法可解决需长期规划的复杂问题并泛化至更大环境。
- 提出RAMP框架，实现在线学习中策略与模型的正向协同优化。
自动化规划算法需要领域模型来描述每个动作的前置条件和效果，但获取此类模型极为困难。虽然已有学习领域模型的算法，但在包含离散与数值状态变量的数值规划环境中，学习模型并结合规划是否有效仍不明确。本文以Minecraft中的两个任务为案例研究，探索学习数值领域模型的收益，并与无模型方案对比。在离线设置下，使用NSAM算法从专家轨迹中学习领域模型，并结合数值规划器求解新问题，称为NSAM_(+p)，其在复杂长程任务上表现优于多种模仿学习与离线强化学习算法，具备更强泛化能力。在在线设置下，提出RAMP框架，利用智能体执行过程中的观测同时训练强化学习策略与学习动作模型，形成策略与模型间的正反馈循环。实验表明，RAMP能生成更高效计划，解决更多问题，优于多个强化学习基线。
原文摘要 · Abstract (English)
Automated Planning algorithms require a model of the domain that specifies the preconditions and effects of each action. Obtaining such a domain model is notoriously hard. Algorithms for learning domain models exist, yet it remains unclear whether learning a domain model and planning is an effective approach for numeric planning environments, i.e., where states include discrete and numeric state variables. In this work, we explore the benefits of learning a numeric domain model and compare it with alternative model-free solutions. As a case study, we use two tasks in Minecraft, a popular sandbox game that has been used as an AI challenge. First, we consider an offline learning setting, where a set of expert trajectories are available to learn from. This is the standard setting for learning domain models. We used the Numeric Safe Action Model Learning (NSAM) algorithm to learn a numeric domain model and solve new problems with the learned domain model and a numeric planner. We call this model-based solution NSAM_(+p), and compare it to several model-free Imitation Learning (IL) and Offline Reinforcement Learning (RL) algorithms. Empirical results show that some IL algorithms can learn faster to solve simple tasks, while NSAM_(+p) allows solving tasks that require long-term planning and enables generalizing to solve problems in larger environments. Then, we consider an online learning setting, where learning is done by moving an agent in the environment. For this setting, we introduce RAMP. In RAMP, observations collected during the agent's execution are used to simultaneously train an RL policy and learn a planning domain action model. This forms a positive feedback loop between the RL policy and the learned domain model. We demonstrate experimentally the benefits of using RAMP, showing that it finds more efficient plans and solves more problems than several RL baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。