提出新方法，让智能体在无奖励情况下高效探索环境。
SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimation
- 基于平稳分布修正估计，直接优化状态熵
- 在离线数据上实现最优状态熵，提升探索效率
- 适合需要高效预训练的强化学习下游任务
在无监督强化学习预训练中，智能体需在不依赖特定任务奖励函数的情况下学习先验策略。本文聚焦于状态熵最大化（SEM），目标是学习使状态平稳分布熵最大的策略。提出SEMDICE，一种严格的离线策略算法，可从任意离线策略数据集直接计算出一个状态熵最大化的平稳马尔可夫策略。该方法在平稳分布空间内直接优化策略，仅需单次计算即可获得最优熵策略。实验表明，SEMDICE在最大化状态熵方面优于基线方法，并在基于SEM的无监督强化学习预训练中展现出最佳的下游任务适应效率。
原文摘要 · Abstract (English)
In the unsupervised pre-training for reinforcement learning, the agent aims to learn a prior policy for downstream tasks without relying on task-specific reward functions. We focus on state entropy maximization (SEM), where the goal is to learn a policy that maximizes the entropy of the state stationary distribution. In this paper, we introduce SEMDICE, a principled off-policy algorithm that computes an SEM policy from an arbitrary off-policy dataset, which optimizes the policy directly within the space of stationary distributions. SEMDICE computes a single, stationary Markov state-entropy-maximizing policy from an arbitrary off-policy dataset. Experimental results demonstrate that SEMDICE outperforms baseline algorithms in maximizing state entropy while achieving the best adaptation efficiency for downstream tasks among SEM-based unsupervised RL pre-training methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。