递归分解轨迹，解决长时序离线强化学习的值函数误差累积问题。
Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL
- 将轨迹分治为平衡二叉树，自底向上更新值函数，避免错误传播。
- 在五个高难度长时序任务上，平均得分从55提升至64，超越所有基线。
- 适合长序列决策、离线强化学习场景，尤其适用于数据有限的任务。
将离线目标条件强化学习（GCRL）扩展到长时序任务面临两大挑战：(1) 长程值学习依赖短程估计，而后者可能仍不准确；(2) 基于最大值的值函数更新会通过反复传播放大过估计。我们提出DCRL（分治强化学习），将每条轨迹段递归分解为平衡二叉树，并从叶子向根训练值函数。每个父节点仅在子节点更新后更新，使用观测路径的精确分解，而非在噪声选项中选择。由于该目标沿演示路径学习值函数，而非最优路径，DCRL联合传播值函数以发现更短路径。得益于平衡二叉树结构，最坏情况下的自举深度从线性降至对数级，实证表明误差累积显著减缓。在多种目标到达任务中，DCRL显著优于以往扁平化离线GCRL方法；在五个最具挑战性的长时序OGBench任务上，其平均得分从55提升至64，超过所有扁平与层级基线。
原文摘要 · Abstract (English)
Scaling offline goal-conditioned reinforcement learning (GCRL) to long-horizon tasks is difficult because (1) long-range value learning depends on shorter-range estimates that may still be inaccurate, and (2) max-based value backups can amplify overestimation through repeated propagation. We propose DCRL (Divide-and-Conquer RL), which recursively decomposes each trajectory segment into a balanced binary tree and trains the values from leaves to root. Each parent is therefore updated only after its children, using an exact factorization of the observed route rather than selecting among noisy alternatives. Since this objective learns values along demonstrated routes that are not necessarily optimal, DCRL jointly propagates values across trajectories to discover shorter routes. Thanks to the balanced binary tree, DCRL reduces worst-case bootstrap depth from linear to logarithmic, and this shorter dependency structure empirically corresponds to much slower error accumulation. Across diverse goal-reaching tasks, DCRL substantially outperforms prior flat offline GCRL methods, and on the five most challenging long-horizon OGBench tasks, it improves the best prior average score from 55 to 64, surpassing all flat and hierarchical baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。