提出分治式价值学习算法，高效解决长程目标导向强化学习问题。
Transitive RL: Value Learning via Divide and Conquer
- 基于分治思想重构价值更新规则，利用目标间三角不等式结构
- 在长轨迹任务中仅需O(log T)次递归，显著降低偏差积累
- 适合高难度长时序任务，优于传统TD与蒙特卡洛方法
本文提出一种新型离线目标条件强化学习价值学习算法——可传递强化学习（Transitive RL, TRL），其核心为分治范式。该方法利用目标间固有的三角不等式结构，构建可实践的分治式价值更新规则。相比传统时序差分（TD）方法，TRL在处理长度为T的轨迹时，理论上只需O(log T)次递归（而TD需O(T)次），有效缓解偏差累积；不同于蒙特卡洛方法，其通过动态规划机制降低方差。实验表明，在多个高难度、长时序基准任务中，TRL性能超越以往所有离线目标条件强化学习算法。
原文摘要 · Abstract (English)
In this work, we present Transitive Reinforcement Learning (TRL), a new value learning algorithm based on a divide-and-conquer paradigm. TRL is designed for offline goal-conditioned reinforcement learning (GCRL) problems, where the aim is to find a policy that can reach any state from any other state in the smallest number of steps. TRL converts a triangle inequality structure present in GCRL into a practical divide-and-conquer value update rule. This has several advantages compared to alternative value learning paradigms. Compared to temporal difference (TD) methods, TRL suffers less from bias accumulation, as in principle it only requires $O(\log T)$ recursions (as opposed to $O(T)$ in TD learning) to handle a length-$T$ trajectory. Unlike Monte Carlo methods, TRL suffers less from high variance as it performs dynamic programming. Experimentally, we show that TRL achieves the best performance in highly challenging, long-horizon benchmark tasks compared to previous offline GCRL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。