通过共享嵌入空间让教师策略更可模仿，减少模仿差距。
Teacher-Student Representational Alignment for Reinforcement Learning-Driven Imitation Learning
- 用对比学习构建共享嵌入空间，隐藏教师独有信息。
- 在多个任务上学生性能提升，模仿差距显著缩小。
- 适合需要高可模仿性的机器人控制场景。
从基于状态的强化学习策略中进行模仿学习，是应对复杂高维观测空间下维度灾难的常见方法。本文针对教师与学生独立训练时产生的不可消除模仿差距问题展开研究，该差距源于教师可依赖学生无法推断的特权状态信息。我们提出一种新算法，通过构建共享嵌入空间，隐藏代理特有观测信息，从而从构造上训练出可模仿的教师策略。该嵌入空间采用自监督对比学习并行训练，通过限制梯度更新编码器网络来防止其提取私有信息。在多个典型任务上的评估表明，相比现有最先进基线，本方法显著降低了模仿差距，提升了学生表现。
原文摘要 · Abstract (English)
Imitation learning (IL) from a state-based reinforcement learning (RL) policy is a common approach to overcome the curse of dimensionality in complex and high-dimensional observation spaces prevalent in robotics. This paper addresses the irreducible imitation gap that emerges when teacher and student are learned in isolation, and the teacher policy has the liberty to rely on privileged state information that the student cannot infer from its observations. Instead of improving poor student performance with RL finetuning after IL, which often requires a whole new training setup, we propose a novel algorithm which learns a shared embedding space that hides agent-specific observations and thus trains imitable teacher policies by construction. We train the shared embedding space with self-supervised contrastive learning in parallel to the teacher policy and prevent it from extracting private information by limiting its gradients from updating the encoder networks. We perform evaluations on several example domains and compare to state-of-the-art baselines showing that our algorithm enables higher student performance with substantially reduced imitation gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。