arXiv:2410.11251cs.LGcs.RO2024-10NeurIPS被引 20

提出新方法让智能体自动学出可复用的独立技能,提升任务解决效率。

Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning

  • 通过互信息约束学习解耦技能,每项技能只控制环境一个因素
  • 在复杂环境中学到的技能能显著提升下游任务的解决效果
  • 适合做分层强化学习、需要模块化技能的场景

智能体从与环境的无监督交互中学习可复用技能是智能的关键特征。然而,现有无监督技能发现方法常产生耦合技能,即一个技能变量同时影响多个环境要素,导致后续技能串联极为困难。本文提出解耦无监督技能发现(DUSDi),将技能分解为独立组件,每个组件仅影响状态空间的一个因子。这些组件可并行组合生成底层动作,并通过分层强化学习高效串联以解决下游任务。DUSDi定义了一种基于互信息的新目标函数,强制不同技能组件的影响解耦,并利用价值因子化高效优化该目标。在一系列挑战性环境中评估表明,DUSDi成功学习到解耦技能,在应用已学技能解决下游任务时显著优于以往方法。代码与技能可视化见 jiahenghu.github.io/DUSDi-site/。

原文摘要 · Abstract (English)

A hallmark of intelligent agents is the ability to learn reusable skills purely from unsupervised interaction with the environment. However, existing unsupervised skill discovery methods often learn entangled skills where one skill variable simultaneously influences many entities in the environment, making downstream skill chaining extremely challenging. We propose Disentangled Unsupervised Skill Discovery (DUSDi), a method for learning disentangled skills that can be efficiently reused to solve downstream tasks. DUSDi decomposes skills into disentangled components, where each skill component only affects one factor of the state space. Importantly, these skill components can be concurrently composed to generate low-level actions, and efficiently chained to tackle downstream tasks through hierarchical Reinforcement Learning. DUSDi defines a novel mutual-information-based objective to enforce disentanglement between the influences of different skill components, and utilizes value factorization to optimize this objective efficiently. Evaluated in a set of challenging environments, DUSDi successfully learns disentangled skills, and significantly outperforms previous skill discovery methods when it comes to applying the learned skills to solve downstream tasks. Code and skills visualization at jiahenghu.github.io/DUSDi-site/.

技能发现强化学习解耦表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。