用随机微分方程建模连续强化学习，揭示神经网络训练中的动态演化规律。
From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
- 将连续强化学习视为连续时间随机过程，引入双时间尺度分析框架。
- 在无限宽两层网络下，首次推导出状态分布的微小变化方程。
- 理论与实验结合，适用于研究过参数化神经网络的训练动态。
我们提出一种新的理论框架，将连续环境下的深度强化学习建模为连续时间随机过程，借鉴随机控制思想。基于前期工作，构建了一个包含探索与随机转移的可行演员-评论家算法模型。针对单隐层神经网络，我们发现环境状态可被表述为两个时间尺度过程：环境时间与梯度时间。在此框架下，我们在两层网络无限宽极限下，刻画了代表环境状态和累积折扣回报估计的时间依赖随机变量在梯度步中的演化。利用随机微分方程理论，首次在连续强化学习中推导出在极小学习率下每步梯度更新时状态分布的微小变化方程。整体工作为研究过参数化神经演员-评论家算法提供了新颖的非参数化形式。我们通过一个简单的连续控制任务实证验证了理论结果。
原文摘要 · Abstract (English)
We present a novel theoretical framework for deep reinforcement learning (RL) in continuous environments by modeling the problem as a continuous-time stochastic process, drawing on insights from stochastic control. Building on previous work, we introduce a viable model of actor-critic algorithm that incorporates both exploration and stochastic transitions. For single-hidden-layer neural networks, we show that the state of the environment can be formulated as a two time scale process: the environment time and the gradient time. Within this formulation, we characterize how the time-dependent random variables that represent the environment's state and estimate of the cumulative discounted return evolve over gradient steps in the infinite width limit of two-layer networks. Using the theory of stochastic differential equations, we derive, for the first time in continuous RL, an equation describing the infinitesimal change in the state distribution at each gradient step, under a vanishingly small learning rate. Overall, our work provides a novel nonparametric formulation for studying overparametrized neural actor-critic algorithms. We empirically corroborate our theoretical result using a toy continuous control task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。