arXiv:2607.20834cs.LG2026-07

通过分层动作分块,提升离线强化学习在长时序任务中的表现。

Offline RL with Hierarchical Action Chunking

论文配图:Offline RL with Hierarchical Action Chunking
图 1 · 摘自论文原文
  • 高阶潜空间规划+低阶动作分块,双层压缩任务时序
  • 在OGBench上整体性能最优,长程导航任务提升显著
  • 适合需要长期规划的复杂决策场景

离线目标条件强化学习有望从静态数据集中学习通用策略。然而,由于时序过长导致的值函数估计误差累积问题,将其扩展到长时序任务仍具挑战。现有分层方法虽通过分解任务为子目标缓解该问题,但常依赖易短视、有偏的底层控制器。本文提出分层隐式Q-分块(HiQC),结合高阶潜空间规划与低阶动作分块。通过将低阶评判器条件于时序扩展的动作序列,HiQC实现无偏的k步值备份,在规划与执行层面同时压缩时序。理论证明,在每步备份误差有界模型下,该双层分解带来更紧的值误差上界,优于标准分层或单一分块。实验显示,HiQC在OGBench基准上整体性能最高,尤其在长时序导航任务如humanoid-giant中增益最大。

原文摘要 · Abstract (English)

Offline goal-conditioned reinforcement learning (RL) holds the promise of learning general-purpose policies from static datasets. However, scaling these methods to long-horizon tasks remains a challenge due to the curse of horizon, where value estimation errors can compound through long chains of bootstrapped Bellman backups. Existing hierarchical approaches mitigate this by decomposing tasks into subgoals, yet they often rely on low-level controllers that suffer from myopic execution and biased value estimates. In this work, we propose Hierarchical Implicit Q-Chunking (HiQC), an offline goal-conditioned RL algorithm that combines high-level latent planning with low-level action chunking. By conditioning the low-level critic on temporally extended action sequences, HiQC enables unbiased k-step value backups, compressing the horizon at both the planning and execution levels. We theoretically demonstrate that this dual decomposition results in a tighter bound on value error under a bounded per-backup error model compared to standard hierarchy or flat chunking alone. Empirically, HiQC achieves the highest aggregate performance among the compared methods on the OGBench suite, with its largest gains on long-horizon navigation tasks such as humanoid-giant.

强化学习离线学习分层决策长时序任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。