让机器人从少量示范中学会多种运动行为，提升真实场景适应力。
Diversifying Policy Behaviors with Extrinsic Behavioral Curiosity
- 用外部评分手动奖励新颖行为，激发策略多样性
- 在人形机器人上超越专家表现20%，最高提升185%
- 适用于多种强化学习框架，适合机器人控制研究者
模仿学习在机器人运动等任务中表现优异，但通常只能学习单一专家策略，限制了行为多样性和现实场景的鲁棒性。为此，本文提出质量多样性逆强化学习（QD-IRL），结合质量-多样性优化与逆强化学习，使智能体能从有限示范中学习多样化行为。引入外源行为好奇心（EBC），通过外部评判器根据行为与大型行为库的差异度提供额外好奇心奖励。在多个机器人运动任务上验证，EBC使基于GAIL、VAIL和DiffAIL的QD-IRL方法性能提升最高达185%、42%和150%，在人形机器人任务中甚至超过专家表现20%。此外，EBC可扩展至基于梯度树结构的质量多样性强化学习算法，显著提升性能，是一种通用的行为多样化策略学习技术。代码已开源：https://github.com/vanzll/EBC。
原文摘要 · Abstract (English)
Imitation learning (IL) has shown promise in various applications (e.g. robot locomotion) but is often limited to learning a single expert policy, constraining behavior diversity and robustness in unpredictable real-world scenarios. To address this, we introduce Quality Diversity Inverse Reinforcement Learning (QD-IRL), a novel framework that integrates quality-diversity optimization with IRL methods, enabling agents to learn diverse behaviors from limited demonstrations. This work introduces Extrinsic Behavioral Curiosity (EBC), which allows agents to receive additional curiosity rewards from an external critic based on how novel the behaviors are with respect to a large behavioral archive. To validate the effectiveness of EBC in exploring diverse locomotion behaviors, we evaluate our method on multiple robot locomotion tasks. EBC improves the performance of QD-IRL instances with GAIL, VAIL, and DiffAIL across all included environments by up to 185%, 42%, and 150%, even surpassing expert performance by 20% in Humanoid. Furthermore, we demonstrate that EBC is applicable to Gradient-Arborescence-based Quality Diversity Reinforcement Learning (QD-RL) algorithms, where it substantially improves performance and provides a generic technique for learning behavioral-diverse policies. The source code of this work is provided at https://github.com/vanzll/EBC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。