将神经网络分解为可复用子策略，生成能加速学习的选项。
Unveiling Options with Neural Decomposition
- 用分段线性激活函数将策略网络转为树结构，提取子树作为子策略。
- 通过不同循环次数的while-loop将子策略封装为选项，提升泛化能力。
- 在网格世界中验证有效，尤其适合探索困难的任务加速学习。
在强化学习中，智能体常针对特定任务学习策略，却难以迁移到相关任务。本文提出一种算法，通过将马尔可夫决策过程的策略神经网络分解为可复用的子策略，用于合成时序扩展的动作（即选项）。我们考虑使用分段线性激活函数的神经网络，使其可映射为类似斜向决策树的结构；树中每个节点对应输入的函数，每个子树即为主策略的子策略。通过为每个子策略添加不同迭代次数的while-loop，将其转化为选项。由于选项数量庞大，我们设计一种基于最小化均匀策略下Levin损失的筛选机制。在两个探索困难的网格世界任务中，实验表明该方法能有效识别出有用选项，从而显著加速相似但不同的新任务的学习过程。
原文摘要 · Abstract (English)
In reinforcement learning, agents often learn policies for specific tasks without the ability to generalize this knowledge to related tasks. This paper introduces an algorithm that attempts to address this limitation by decomposing neural networks encoding policies for Markov Decision Processes into reusable sub-policies, which are used to synthesize temporally extended actions, or options. We consider neural networks with piecewise linear activation functions, so that they can be mapped to an equivalent tree that is similar to oblique decision trees. Since each node in such a tree serves as a function of the input of the tree, each sub-tree is a sub-policy of the main policy. We turn each of these sub-policies into options by wrapping it with while-loops of varied number of iterations. Given the large number of options, we propose a selection mechanism based on minimizing the Levin loss for a uniform policy on these options. Empirical results in two grid-world domains where exploration can be difficult confirm that our method can identify useful options, thereby accelerating the learning process on similar but different tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。