用图搜索替代高层策略，高效拼接跨轨迹状态转移。
Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
- 将子目标选择转为图搜索问题，通过时序距离嵌入聚类状态
- 在关键任务中达88.3分，远超旧方法1.0的极限
- 适合需要长程规划与状态复用的强化学习场景
现有离线层级强化学习方法依赖高层策略生成子目标序列，但随着任务时域增长，效率下降，且缺乏有效策略来拼接不同轨迹中的有用状态转移。本文提出图辅助拼接（GAS）框架，将子目标选择建模为图搜索问题，而非显式学习高层策略。通过将状态嵌入时序距离表示（TDR）空间，GAS将来自不同轨迹的语义相似状态聚类为统一图节点，实现高效状态转移拼接。随后使用最短路径算法在图中选取子目标序列，低层策略则学习到达这些子目标。为提升图质量，引入时序效率（TE）度量，过滤噪声或低效状态转移，显著提升任务性能。GAS在行走、导航和操作任务中均优于先前离线HRL方法。尤其在最需拼接的任务中，取得88.3分，大幅超越此前最优的1.0分。源代码已开源。
原文摘要 · Abstract (English)
Existing offline hierarchical reinforcement learning methods rely on high-level policy learning to generate subgoal sequences. However, their efficiency degrades as task horizons increase, and they lack effective strategies for stitching useful state transitions across different trajectories. We propose Graph-Assisted Stitching (GAS), a novel framework that formulates subgoal selection as a graph search problem rather than learning an explicit high-level policy. By embedding states into a Temporal Distance Representation (TDR) space, GAS clusters semantically similar states from different trajectories into unified graph nodes, enabling efficient transition stitching. A shortest-path algorithm is then applied to select subgoal sequences within the graph, while a low-level policy learns to reach the subgoals. To improve graph quality, we introduce the Temporal Efficiency (TE) metric, which filters out noisy or inefficient transition states, significantly enhancing task performance. GAS outperforms prior offline HRL methods across locomotion, navigation, and manipulation tasks. Notably, in the most stitching-critical task, it achieves a score of 88.3, dramatically surpassing the previous state-of-the-art score of 1.0. Our source code is available at: https://github.com/qortmdgh4141/GAS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。