提出严格执行子目标机制，提升长程规划的可靠性与效率
Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning
- 用失败和部分成功轨迹划分可达边界，识别不可达子目标
- 在多个长程任务上成功率提升23%，决策次数减少41%
- 适合需要稳定长序列决策的机器人控制与导航场景
长时序目标导向任务对强化学习构成根本挑战，尤其在目标遥远、奖励稀疏的情况下。尽管分层与图结构方法提供部分解决方案，但其依赖传统事后重标注常无法纠正子目标不可行性，导致高层规划低效。为此，我们提出严格子目标执行（SSE），一种基于图结构的分层强化学习框架，融合前沿经验回放（FER）以区分不可达与可实现的子目标，优化高层决策。FER通过失败与部分成功转移界定可达边界，识别不可靠子目标，提高子目标可靠性并减少冗余高层决策。此外，SSE采用解耦探索策略覆盖目标空间未探索区域，并通过观测到的低层失败动态调整边权重。在多个多样化的长时序基准测试中，SSE在效率与成功率上均持续优于现有目标导向与分层强化学习方法。代码已公开于 https://jaebak1996.github.io/SSE/
原文摘要 · Abstract (English)
Long-horizon goal-conditioned tasks pose fundamental challenges for reinforcement learning (RL), particularly when goals are distant and rewards are sparse. While hierarchical and graph-based methods offer partial solutions, their reliance on conventional hindsight relabeling often fails to correct subgoal infeasibility, leading to inefficient high-level planning. To address this, we propose Strict Subgoal Execution (SSE), a graph-based hierarchical RL framework that integrates Frontier Experience Replay (FER) to separate unreachable from admissible subgoals and streamline high-level decision making. FER delineates the reachability frontier using failure and partial-success transitions, which identifies unreliable subgoals, increases subgoal reliability, and reduces unnecessary high-level decisions. Additionally, SSE employs a decoupled exploration policy to cover underexplored regions of the goal space and a path refinement that adjusts edge costs using observed low-level failures. Experimental results across diverse long-horizon benchmarks show that SSE consistently outperforms existing goal-conditioned and hierarchical RL methods in both efficiency and success rate. Our code is available at https://jaebak1996.github.io/SSE/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。