通过可度量的任务表征学习,自动生成更优的强化学习课程。
Curriculum reinforcement learning with measurable task representation learning

- 用变分自编码器将任务映射到隐空间,实现任务相似性度量。
- 在复杂导航任务中优于基于插值和GAN的现有方法。
- 适合需要自动课程设计的复杂强化学习场景。
在课程强化学习(CRL)中,智能体逐步积累一系列任务的知识,最终解决具有挑战性的目标任务。早期工作关注任务序列的设计,近期研究转向自动课程生成。主流的基于插值的CRL范式通过有意义的距离度量,在任务空间中对初始任务分布与目标任务分布进行插值,自动生成中间任务。然而,在复杂导航任务中,非欧几里得的任务空间使该假设失效。为此,本文提出一种基于可度量任务表征学习的新型自动课程生成方法。通过变分自编码器结构,编码奖励与状态转移信息,将任务空间转换至隐空间,获得具备任务相似性度量特性的隐任务表征,使相近的任务嵌入对应于奖励和状态转移相似的任务。基于学习到的任务表征,进一步构建自动课程生成方案,能逐步生成越来越接近目标任务的新任务。在多种复杂导航任务上的实验表明,该方法优于基于插值和生成对抗网络的先进CRL方法。
原文摘要 · Abstract (English)
In curriculum reinforcement learning (CRL), an agent incrementally accumulates knowledge over a sequence of tasks (i.e., a curriculum), and the learning process is aimed at using the accumulated knowledge to finally solve a challenging target task. While early CRL works focus on sequencing candidate tasks, recent research explores automatic curriculum generation. Among the rich CRL literature, the interpolation-based CRL paradigm is a main body, which automatically generates intermediate tasks by interpolating between the initial task distribution and the target task distribution in task space with meaningful distance metrics (i.e., can measure the task similarity). However, in challenging navigation tasks, the non-Euclidean context (task) space invalidates this assumption. To achieve automatic curriculum generation in complex task, we propose a novel automatic curriculum generation approach based on measurable task representation learning. To better measure the similarity, we propose to transform the task space to a latent space. Through a variational autoencoder structure that encodes the reward and the state transitions, we achieve a latent task representation with a task similarity measurement property, and two close task embeddings correspond to two similar tasks in terms of rewards and state transitions. Based on the learned task representation, we further develop an automatic curriculum generation scheme, which can effectively generate new tasks more and more similar to the target task. We evaluate our method in a variety of challenging navigation tasks, and the experiment results indicate that the proposed approach surpasses state-of-the-art CRL approaches based on interpolation and generative adversarial networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。