arXiv:2602.01619cs.LGcs.AI2026-02

通过分解状态空间发现更复杂多样的无监督技能。

SUSD: Structured Unsupervised Skill Discovery through State Factorization

  • 将环境状态分解为独立因素,为每个因素分配专属技能变量。
  • 在1到10个因素的环境中显著优于现有方法,发现更丰富技能。
  • 适合需要精细控制和组合任务的强化学习研究者。

无监督技能发现(USD)旨在不依赖外部奖励的情况下自主学习多样技能。现有基于互信息的方法倾向于生成简单静态技能,而距离最大化方法虽提升动态性,仍难覆盖所有可控因素。本文提出SUSD框架,通过将状态空间分解为独立组件(如物体或可控实体),为不同因素分配专属技能变量,实现更细粒度的技能控制。同时引入动态模型追踪各因素学习进度,自适应引导智能体关注未充分探索的因素。该结构化方法不仅促进发现更丰富多样的技能,还生成因子解耦的技能表示,支持通过分层强化学习高效训练组合式下游任务。在包含1至10个因素的三个环境上的实验表明,SUSD无需监督即可发现复杂技能,显著超越现有无监督方法。代码已公开:https://github.com/hadi-hosseini/SUSD。

原文摘要 · Abstract (English)

Unsupervised Skill Discovery (USD) aims to autonomously learn a diverse set of skills without relying on extrinsic rewards. One of the most common USD approaches is to maximize the Mutual Information (MI) between skill latent variables and states. However, MI-based methods tend to favor simple, static skills due to their invariance properties, limiting the discovery of dynamic, task-relevant behaviors. Distance-Maximizing Skill Discovery (DSD) promotes more dynamic skills by leveraging state-space distances, yet still fall short in encouraging comprehensive skill sets that engage all controllable factors or entities in the environment. In this work, we introduce SUSD, a novel framework that harnesses the compositional structure of environments by factorizing the state space into independent components (e.g., objects or controllable entities). SUSD allocates distinct skill variables to different factors, enabling more fine-grained control on the skill discovery process. A dynamic model also tracks learning across factors, adaptively steering the agent's focus toward underexplored factors. This structured approach not only promotes the discovery of richer and more diverse skills, but also yields a factorized skill representation that enables fine-grained and disentangled control over individual entities which facilitates efficient training of compositional downstream tasks via Hierarchical Reinforcement Learning (HRL). Our experimental results across three environments, with factors ranging from 1 to 10, demonstrate that our method can discover diverse and complex skills without supervision, significantly outperforming existing unsupervised skill discovery methods in factorized and complex environments. Code is publicly available at: https://github.com/hadi-hosseini/SUSD.

无监督学习技能发现分层强化学习状态分解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。