提出一种高效探索策略，实现可控马尔可夫链的最优学习。
An Optimal Policy for Learning Controllable Dynamics by Exploration
- 基于随时间变化的控制集，贪婪地最大化信息增益。
- 揭示瞬态、吸收态等状态导致非平稳策略必要性。
- 适用于需高效探索的强化学习与最优控制场景。
可控马尔可夫链描述了序列决策任务的动力学，是最佳控制与强化学习的核心组件。本文给出了在有限时间窗内通过探索学习未知环境可控动力学的最优策略的一般形式。该策略实现简单、计算高效，使智能体在探索过程中通过从随时间变化的约束控制集中选择动作，以贪心方式最大化信息增益。我们提出了控制集的简单参数化方法，并给出求解最优策略的算法。策略存在的原因在于某些状态（如瞬态、吸收态、不可回溯状态）限制了动力学控制；这些状态的存在使得非平稳策略对实现最优探索至关重要。本文详细分析了六类有趣的可控动力学实例。通过计数论证、与次优策略对比以及动态规划中的序列改进性质，证明了策略的最优性。
原文摘要 · Abstract (English)
Controllable Markov chains describe the dynamics of sequential decision making tasks and are the central component in optimal control and reinforcement learning. In this work, we give the general form of an optimal policy for learning controllable dynamics in an unknown environment by exploring over a limited time horizon. This policy is simple to implement and efficient to compute, and allows an agent to ``learn by exploring" as it maximizes its information gain in a greedy fashion by selecting controls from a constraint set that changes over time during exploration. We give a simple parameterization for the set of controls, and present an algorithm for finding an optimal policy. The reason for this policy is due to the existence of certain types of states that restrict control of the dynamics; such as transient states, absorbing states, and non-backtracking states. We show why the occurrence of these states makes a non-stationary policy essential for achieving optimal exploration. Six interesting examples of controllable dynamics are treated in detail. Policy optimality is demonstrated using counting arguments, comparing with suboptimal policies, and by making use of a sequential improvement property from dynamic programming.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。