用线性回归和拟牛顿法改进连续动作深度强化学习，提升稳定性与性能。
Application of linear regression and quasi-Newton methods to the deep reinforcement learning in continuous action cases
- 将线性回归与DDPG结合，用拟牛顿法求最优动作
- 在MuJoCo环境中部分任务性能优于传统方法
- 适合研究连续动作策略优化的算法设计者
线性回归（LR）虽表达能力有限，但参数求解简便。为提升深度强化学习性能，Levine等人提出将深度Q网络（DQN）与线性回归结合的最小二乘深度Q网络（LS-DQN），但该方法仅适用于离散动作。本文提出双最小二乘深度确定性策略梯度（DLS-DDPG）方法，用于处理连续动作场景。该方法将线性回归与深度确定性策略梯度（DDPG）结合，其评论家网络更新采用类似拟合Q迭代的算法；同时，利用拟牛顿法计算最优动作，既作为智能体执行动作，也作为演员网络线性回归更新的训练数据。在MuJoCo环境中的数值实验表明，所提方法在部分任务中性能有所提升，但仍面临正则化项难以减小等挑战。
原文摘要 · Abstract (English)
The linear regression (LR) method offers the advantage that optimal parameters can be calculated relatively easily, although its representation capability is limited than that of the deep learning technique. To improve deep reinforcement learning, the Least Squares Deep Q Network (LS-DQN) method was proposed by Levine et al., which combines Deep Q Network (DQN) with LR method. However, the LS-DQN method assumes that the actions are discrete. In this study, we propose the Double Least Squares Deep Deterministic Policy Gradient (DLS-DDPG) method to address this limitation. This method combines the LR method with the Deep Deterministic Policy Gradient (DDPG) technique, one of the representative deep reinforcement learning algorithms for continuous action cases. For the LR update of the critic network, DLS-DDPG uses an algorithm similar to the Fitted Q iteration, the method which LS-DQN adopted. In addition, we calculated the optimal action using the quasi-Newton method and used it as both the agent's action and the training data for the LR update of the actor network. Numerical experiments conducted in MuJoCo environments showed that the proposed method improved performance at least in some tasks, although there are difficulties such as the inability to make the regularization terms small.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。