用自回归生成子目标序列，提升长程离线目标导向强化学习性能
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
- 将决策过程建模为自回归序列生成，逐步产生潜变量子目标
- 在多个导航与操作任务上优于主流基线，长程任务表现显著提升
- 适合需要复杂规划的离线强化学习场景，如机器人路径规划
离线目标条件强化学习在长时程任务中仍具挑战性。尽管分层方法通过任务分解缓解此问题，但多数现有方法依赖独立的高层与低层网络，且仅生成单一中间子目标，导致长程决策结构受限。为此，我们受思维链推理启发，提出链式目标分层策略（CoGHP），将分层决策重构为统一架构中的自回归序列建模。给定状态与最终目标，CoGHP 自回归生成一系列潜变量子目标，随后输出原始动作；每个子目标作为推理步骤，指导后续预测。为高效实现，引入 MLP-Mixer 主干，支持跨标记通信，捕捉状态、目标、潜变量子目标与动作间的结构关系。在多个具有挑战性的导航与操作基准测试中，CoGHP 持续超越强基线，在长时程任务上表现更优。
原文摘要 · Abstract (English)
Offline goal-conditioned reinforcement learning remains challenging for long-horizon tasks. While hierarchical approaches mitigate this issue by decomposing tasks, most existing methods rely on separate high- and low-level networks and generate only a single intermediate subgoal, leaving several structural limitations in long-horizon decision-making. To address this limitation, we draw inspiration from chain-of-thought reasoning and propose the Chain-of-Goals Hierarchical Policy (CoGHP), a novel framework that reformulates hierarchical decision-making as autoregressive sequence modeling within a unified architecture. Given a state and a final goal, CoGHP autoregressively generates a sequence of latent subgoals followed by the primitive action, where each latent subgoal acts as a reasoning step that conditions subsequent predictions. To implement this efficiently, we introduce an MLP-Mixer backbone, which supports cross-token communication and captures structural relationships among state, goal, latent subgoals, and action. Across challenging navigation and manipulation benchmarks, CoGHP consistently outperforms strong offline baselines, demonstrating improved performance on long-horizon tasks. Project page: https://wlsdn9350.github.io/projects/coghp/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。