用离散可达性替代连续距离，提升长程视觉规划成功率
DHP: Discrete Hierarchical Planning for Hierarchical Reinforcement Learning Agents
- 以离散可达性判断子目标可行性，递归构建树状计划
- 25房间导航任务达100%成功率，较基线提升10个百分点
- 无需专家数据即可生成训练样本，适合复杂控制任务
层次强化学习（HRL）代理在长程视觉规划中常因依赖易出错的距离度量而受阻。我们提出离散层次规划（DHP），用离散可达性检查替代连续距离估计，评估子目标可行性。DHP通过将长期目标分解为一系列简单子任务，递归构建树状计划，并采用新颖的优势估计策略，天然奖励更短计划，且可泛化至训练深度之外。为解决数据效率问题，引入一种探索策略，无需专家数据即可为目标规划模块生成针对性训练样本。在25房间导航环境中实验表明，成功率达100%（对比基线90%）。此外，离线版本在OGBench基准上取得领先结果，在giant HumanoidMaze任务上绝对提升高达71%。核心贡献与架构无关。方法还适用于基于动量的控制任务，重规划仅需log N步。理论分析与消融实验验证设计合理性。
原文摘要 · Abstract (English)
Hierarchical Reinforcement Learning (HRL) agents often struggle with long-horizon visual planning due to their reliance on error-prone distance metrics. We propose Discrete Hierarchical Planning (DHP), a method that replaces continuous distance estimates with discrete reachability checks to evaluate subgoal feasibility. DHP recursively constructs tree-structured plans by decomposing long-term goals into sequences of simpler subtasks, using a novel advantage estimation strategy that inherently rewards shorter plans and generalizes beyond training depths. In addition, to address the data efficiency challenge, we introduce an exploration strategy that generates targeted training examples for the planning modules without needing expert data. Experiments in 25-room navigation environments demonstrate a 100% success rate (vs. 90% baseline). We also present an offline variant that achieves state-of-the-art results on OGBench benchmarks, with up to 71% absolute gains on giant HumanoidMaze tasks, demonstrating our core contributions are architecture-agnostic. The method also generalizes to momentum-based control tasks and requires only log N steps for replanning. Theoretical analysis and ablations validate our design choices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。