让智能体一次性学会所有目标，训练速度提升250倍以上
Goal-Conditioned Agents that Learn Everything All at Once
- 同一网络并行输出所有目标的值和动作，实现高效多目标学习
- 在Craftax任务上显著优于其他方法，连续控制任务也达到领先水平
- 可作为教师网络提升性能，适合复杂环境下的强化学习实践
目标条件化强化学习智能体在探索环境中会获得大量信息，但传统基于当前目标的在线更新会丢弃大部分数据。全目标学习（all-goals learning）通过离线方式利用每个轨迹中的每条经验学习所有目标，能最大化信息提取，但通常因朴素重标注而计算开销巨大。本文提出‘一次学一切’（LEO）方法，通过联合输出所有目标的值与动作，仅需一次网络前向传播即可完成并行的全目标更新，实现>250倍于传统全目标重标注的加速。实验表明，LEO在目标条件化游戏环境Craftax上表现显著优于现有方法，在连续控制任务中也具备竞争力。进一步将LEO用作教师网络时性能更优。代码已开源。
原文摘要 · Abstract (English)
A goal-conditioned reinforcement learning agent exploring an environment will see a wealth of information throughout a trajectory, most of which is discarded when only performing on-policy updates with respect to the commanded goal. All-goals learning, where each transition is used for learning off-policy with respect to every goal, allows agents to extract maximal information, however it is usually computationally infeasible when done via naive relabelling. This can be overcome by jointly outputting values and actions for every goal at once, allowing for efficient, parallel all-goals updates with a single pass through the network, in a process we call Learning Everything all at Once (LEO). We show that this approach significantly outperforms other methods on goal-conditioned Craftax and is competitive with existing baselines on continuous control environments, while achieving a >250x speed-up compared to all-goals relabelling. We then go on to show that this approach can be made even more powerful by using LEO as a teacher network, rather than a direct actor. We hope that, by unlocking all-goals learning at scale, LEO can serve as a useful tool for RL practitioners in complex environments. We open source our code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。