让智能体在线学习动作模型，边试边改，自动规划更高效。
RAMP: Hybrid DRL for Online Learning of Numeric Action Models
- 用强化学习与规划协同训练，边交互边构建动作模型。
- 在标准测试集上比PPO算法成功率更高、计划质量更好。
- 适合需要实时学习和决策的自动化系统研究者。
自动规划算法需要明确的动作模型来描述每个动作的前提和效果，但获取此类模型往往困难。从观测中学习动作模型是可行的，但现有数值域方法均为离线学习，需依赖专家轨迹输入。本文提出一种在线学习数值规划动作模型的新策略——RAMP：通过与环境交互，同时训练深度强化学习（DRL）策略、从历史交互中学习数值动作模型，并在可能时利用该模型进行未来动作规划。三者形成正向反馈：强化学习策略收集数据以优化动作模型，而规划器生成计划继续训练强化学习策略。为促进强化学习与数值规划的集成，我们开发了Numeric PDDLGym，一个将数值规划问题自动转化为Gym环境的框架。在标准IPC数值域上的实验表明，RAMP在可解性和计划质量方面显著优于著名的PPO算法。
原文摘要 · Abstract (English)
Automated planning algorithms require an action model specifying the preconditions and effects of each action, but obtaining such a model is often hard. Learning action models from observations is feasible, but existing algorithms for numeric domains are offline, requiring expert traces as input. We propose the Reinforcement learning, Action Model learning, and Planning (RAMP) strategy for learning numeric planning action models online via interactions with the environment. RAMP simultaneously trains a Deep Reinforcement Learning (DRL) policy, learns a numeric action model from past interactions, and uses that model to plan future actions when possible. These components form a positive feedback loop: the RL policy gathers data to refine the action model, while the planner generates plans to continue training the RL policy. To facilitate this integration of RL and numeric planning, we developed Numeric PDDLGym, an automated framework for converting numeric planning problems to Gym environments. Experimental results on standard IPC numeric domains show that RAMP significantly outperforms PPO, a well-known DRL algorithm, in terms of solvability and plan quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。