用连续时间强化学习解决多状态切换的最优决策问题。
Continuous-time reinforcement learning for optimal switching over multiple regimes
- 通过马尔可夫链生成器矩阵随机化切换时机与状态选择。
- 证明了贝尔曼方程解的存在性，算法迭代收敛且值函数随温度趋零而逼近经典解。
- 适用于金融领域的多状态策略优化，模型无关算法高效可行。
本文研究连续时间强化学习在多状态切换问题中的应用。考虑一种基于熵正则化的探索性建模，其中智能体通过关联的连续时间有限状态马尔可夫链的生成器矩阵，随机决定切换时机和状态选择。建立了相应的哈密顿-雅可比-贝尔曼（HJB）方程组的适定性，并给出了最优策略的刻画。通过分析方程组，严格证明了策略改进和迭代收敛性。此外，证明了探索性形式下的值函数在温度参数趋于零时收敛到经典形式的值函数。最后，基于鞅表征设计并实现了无需模型的强化学习算法。数值实验在金融场景中验证了该算法的有效性和效率。
原文摘要 · Abstract (English)
This paper studies the continuous-time reinforcement learning (RL) for optimal switching problems across multiple regimes. We consider a type of exploratory formulation under entropy regularization where the agent randomizes both the timing of switches and the selection of regimes through the generator matrix of an associated continuous-time finite-state Markov chain. We establish the well-posedness of the associated system of Hamilton-Jacobi-Bellman (HJB) equations and provide a characterization of the optimal policy. The policy improvement and the convergence of the policy iterations are rigorously established by analyzing the system of equations. We also show that the value function in the exploratory formulation converges to the one in the classical formulation as the temperature parameter vanishes. Finally, a model-free reinforcement learning algorithm is devised and implemented by invoking the policy evaluation based on the martingale characterization. Our numerical examples with financial applications illustrate the effectiveness and efficiency of the proposed RL algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。