无监督发现周期性技能，提升机器人运动多样性
Periodic Skill Discovery
- 将状态映射到环形隐空间，天然建模周期性行为
- 在复杂任务中学习多样周期技能，像素输入下表现优异
- 可提升下游任务性能，适合强化学习中的机器人控制
无监督强化学习中的技能发现旨在不依赖外部奖励的情况下学习多样化行为。然而，现有方法常忽视所学技能的周期性，仅关注状态与技能间的相互依赖或潜在空间中的移动距离。考虑到许多机器人任务（尤其是运动类）涉及多时间尺度的周期性行为，发现多样化周期性技能至关重要。为此，我们提出周期性技能发现（PSD）框架，以无监督方式发现周期性行为。PSD的核心思想是训练一个编码器，将状态映射到环形潜在空间，从而在隐表示中自然编码周期性。通过捕捉时间距离，PSD可在复杂机器人任务中有效学习具有不同周期的技能，即使使用基于像素的观测也表现良好。实验表明，这些学习到的技能在后续任务（如跨栏）中表现优异。此外，将PSD与已有技能发现方法结合，能进一步扩大智能体的行为库。代码与演示见 https://jonghaepark.github.io/psd/
原文摘要 · Abstract (English)
Unsupervised skill discovery in reinforcement learning (RL) aims to learn diverse behaviors without relying on external rewards. However, current methods often overlook the periodic nature of learned skills, focusing instead on increasing the mutual dependence between states and skills or maximizing the distance traveled in latent space. Considering that many robotic tasks - particularly those involving locomotion - require periodic behaviors across varying timescales, the ability to discover diverse periodic skills is essential. Motivated by this, we propose Periodic Skill Discovery (PSD), a framework that discovers periodic behaviors in an unsupervised manner. The key idea of PSD is to train an encoder that maps states to a circular latent space, thereby naturally encoding periodicity in the latent representation. By capturing temporal distance, PSD can effectively learn skills with diverse periods in complex robotic tasks, even with pixel-based observations. We further show that these learned skills achieve high performance on downstream tasks such as hurdling. Moreover, integrating PSD with an existing skill discovery method offers more diverse behaviors, thus broadening the agent's repertoire. Our code and demos are available at https://jonghaepark.github.io/psd/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。