用自由能原理发现任务中的关键中间目标，无需预先知道任务结构。
Subgoal Discovery Using a Free Energy Paradigm and State Aggregations
- 通过比较原始状态空间与聚合状态空间的模型变化，识别不可预测状态作为子目标。
- 在网格世界导航任务中成功发现子目标，且在随机环境中仍保持稳定。
- 适合需要自动分解复杂任务的强化学习场景，尤其适用于无先验知识的设置。
强化学习在解决复杂序列决策任务中扮演重要角色。层级化和目标条件化的强化学习是应对样本效率低下与奖励设计困难的有前景方法，它们通过将任务分解为更简单的子任务，并在动作空间上进行时间抽象来实现。这些方法的关键组件之一是子目标发现。我们可利用子目标状态构建动作层次结构并分解复杂任务。本文假设子目标状态具有更高的不可预测性，提出一种基于自由能原理的子目标发现方法。该方法通过自由能选择主状态空间与聚合状态空间之间的切换。模型从邻近状态到某一给定状态的变化程度反映了该状态的不可预测性，因此被用于子目标识别。在网格世界等导航任务上的实验表明，所提方法可在无任务先验知识的情况下实现子目标发现，且对环境随机性具有鲁棒性。
原文摘要 · Abstract (English)
Reinforcement learning (RL) plays a major role in solving complex sequential decision-making tasks. Hierarchical and goal-conditioned RL are promising methods for dealing with two major problems in RL, namely sample inefficiency and difficulties in reward shaping. These methods tackle the mentioned problems by decomposing a task into simpler subtasks and temporally abstracting a task in the action space. One of the key components for task decomposition of these methods is subgoal discovery. We can use the subgoal states to define hierarchies of actions and also use them in decomposing complex tasks. Under the assumption that subgoal states are more unpredictable, we propose a free energy paradigm to discover them. This is achieved by using free energy to select between two spaces, the main space and an aggregation space. The $model \; changes$ from neighboring states to a given state shows the unpredictability of a given state, and therefore it is used in this paper for subgoal discovery. Our empirical results on navigation tasks like grid-world environments show that our proposed method can be applied for subgoal discovery without prior knowledge of the task. Our proposed method is also robust to the stochasticity of environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。