自适应分步细化目标，提升长时序离线强化学习效果
Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
- 从终点反向递归生成中间目标，动态调整细化过程
- 在OGBench上多个长时任务成功率显著提升
- 适合需要分步规划的复杂离线强化学习场景
离线目标条件强化学习在长时序任务中面临挑战：远距离状态-目标对提供弱监督，价值估计易受累积自举误差影响。层次化方法通过引入中间子目标缓解此问题，但固定时间抽象或固定层级深度常与不同可达性跨度的状态-目标对不匹配。本文提出完全离线的目标条件强化学习框架CFHRL，通过自适应细化远端目标来提升执行效率。从最终目标出发，CFHRL递归生成候选中间目标，基于回放数据训练，并在当前目标被评估为局部可执行（由学习到的可达性代价判断）时停止细化。核心思想是子目标无需精确位于中点或全局最优路径上，只需提供可靠进展并降低剩余难度，从而支持后续在更短时序内进一步细化。风格化分析进一步验证了近似递归收缩的鲁棒性。在OGBench上的实验显示，在多个长时任务中取得显著性能提升，消融实验验证了所提细化与停止机制的有效性。
原文摘要 · Abstract (English)
Offline goal-conditioned reinforcement learning (GCRL) is challenging in long-horizon tasks, where distant state--goal pairs provide weak supervision and value estimates become vulnerable to accumulated bootstrapping errors. Hierarchical methods mitigate this difficulty by introducing intermediate subgoals, but fixed temporal abstractions or fixed hierarchy depths can be mismatched to state--goal pairs with different reachability horizons. We propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. Starting from the final goal, CFHRL recursively proposes intermediate targets, trained from replay-supported candidates, and stops refinement once the current target is estimated to be locally executable by a learned reachability cost. The key idea is that a subgoal need not be an exact midpoint or globally optimal waypoint; it only needs to provide reliable progress and reduce the remaining reaching difficulty, enabling subsequent refinement over shorter horizons. A stylized analysis further supports the robustness of approximate recursive contraction. Experiments on OGBench show substantial gains on several long-horizon tasks, with ablations validating the proposed refinement and stopping mechanisms
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。