用分层策略与规划提升机器人多目标强化学习效率
Proposing Hierarchical Goal-Conditioned Policy Planning in Multi-Goal Reinforcement Learning
- 分层使用短周期目标条件策略，结合蒙特卡洛树搜索生成高层动作
- 单个计划树长期保存知识，实现动作复用与未来动作预判
- 适合复杂任务中需要高效探索与规划的机器人学习场景
类人机器人需掌握大量稀疏奖励任务，对强化学习（RL）构成挑战。本文提出一种结合强化学习与自动规划的方法。该方法采用分层组织的短期目标条件策略（GCPs），利用蒙特卡洛树搜索（MCTS）生成高层动作（HLAs）进行规划，而非直接使用原始动作。整个代理生命周期内维护单一计划树，存储目标达成知识。该层次结构提升了样本效率，加速了推理过程，通过重用高层动作和预判未来动作实现。所提出的分层目标条件策略规划（HGCPP）框架首次融合了GCP、MCTS与层次强化学习，有望改善复杂任务中的探索与规划性能。
原文摘要 · Abstract (English)
Humanoid robots must master numerous tasks with sparse rewards, posing a challenge for reinforcement learning (RL). We propose a method combining RL and automated planning to address this. Our approach uses short goal-conditioned policies (GCPs) organized hierarchically, with Monte Carlo Tree Search (MCTS) planning using high-level actions (HLAs). Instead of primitive actions, the planning process generates HLAs. A single plan-tree, maintained during the agent's lifetime, holds knowledge about goal achievement. This hierarchy enhances sample efficiency and speeds up reasoning by reusing HLAs and anticipating future actions. Our Hierarchical Goal-Conditioned Policy Planning (HGCPP) framework uniquely integrates GCPs, MCTS, and hierarchical RL, potentially improving exploration and planning in complex tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。