不需训练，用图搜索让旧策略搞定长程任务
Test-Time Graph Search for Goal-Conditioned Reinforcement Learning
- 用离线数据构建图，动态选子目标规划路径
- 解决价值估计误差，使长程任务成功率从近零升至90%以上
- 轻量无额外训练，适合想提升现有模型的开发者
离线目标条件强化学习在长程任务中常因价值估计误差累积导致策略不可靠。传统认为长期规划需专门训练，但本文表明：将现有GCRL策略与轻量、无需训练的规划封装结合，即可完成长程任务。我们发现标准目标条件价值函数已包含足够局部一致的几何结构用于规划。提出测试时图搜索（TTGS）方法，在离线数据集上构建图并采用自适应子目标选择策略。为缓解最短路径搜索中不可靠价值估计问题，引入软惩罚机制以抑制远距离转移。该方法计算开销极低，无需额外监督或参数更新。在OGBench基准上，TTGS显著提升多种基线学习器与任务的成功率，尤其在挑战性的长程运动任务中，部分成功率从接近零提升至超90%，通常达到或超过需复杂辅助训练的方法。
原文摘要 · Abstract (English)
Offline goal-conditioned reinforcement learning (GCRL) often struggles with long-horizon tasks, where errors in value estimation accumulate and produce unreliable policies. It is typically assumed that effective long-term planning is infeasible without specialized training. In contrast, our work demonstrates that existing GCRL policies can complete long-horizon tasks when combined with a lightweight, training-free planning wrapper. We find that standard goal-conditioned value functions encode locally consistent geometric structure sufficient for planning. Our approach, Test-Time Graph Search (TTGS), constructs a graph over the offline dataset and employs an adaptive subgoal selection strategy. To address unreliable value estimates during shortest-path search, we propose a novel mechanism that softly penalizes long-distance transitions. Our method incurs negligible computational overhead and requires no additional supervision or parameter updates. On the OGBench benchmark, TTGS significantly boosts success rates across multiple base learners and tasks, with primary gains on challenging long-horizon locomotion tasks where some success rates are improved from near-zero to over 90\%, often matching or outperforming methods that require complex auxiliary training. Code and videos can be found at https://ktolnos.github.io/ttgs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。