用卡尔曼滤波快速优化强化学习超参数，提升训练效率与稳定性。
HyperController: A Hyperparameter Controller for Fast and Stable Training of Reinforcement Learning Neural Networks
- 将超参数优化建模为线性高斯动态系统，用卡尔曼滤波高效估计目标函数。
- 在五种环境中的四次实验中，获得最高中位数奖励，优于其他算法。
- 适合需要快速稳定训练的强化学习应用，尤其适合资源受限场景。
我们提出超参数控制器（HyperController），一种计算高效的强化学习神经网络训练期间超参数优化算法。HyperController通过将超参数优化问题建模为未知的线性高斯动态系统（状态线性变化）来实现快速优化，并持续提升强化学习神经网络性能，从而加速训练与部署。它利用卡尔曼滤波学习超参数目标函数的高效表示，该方法是线性高斯动态系统的最优一步预测器。为验证性能，我们在多个OpenAI Gymnasium环境中将其作为超参数优化器应用于强化学习神经网络训练。在五种环境中的四类中，HyperController在评估时实现了最高的中位数奖励，显著优于其他算法。结果表明，HyperController在实现强化学习神经网络高效且稳定训练方面具有巨大潜力。
原文摘要 · Abstract (English)
We introduce Hyperparameter Controller (HyperController), a computationally efficient algorithm for hyperparameter optimization during training of reinforcement learning neural networks. HyperController optimizes hyperparameters quickly while also maintaining improvement of the reinforcement learning neural network, resulting in faster training and deployment. It achieves this by modeling the hyperparameter optimization problem as an unknown Linear Gaussian Dynamical System, which is a system with a state that linearly changes. It then learns an efficient representation of the hyperparameter objective function using the Kalman filter, which is the optimal one-step predictor for a Linear Gaussian Dynamical System. To demonstrate the performance of HyperController, it is applied as a hyperparameter optimizer during training of reinforcement learning neural networks on a variety of OpenAI Gymnasium environments. In four out of the five Gymnasium environments, HyperController achieves highest median reward during evaluation compared to other algorithms. The results exhibit the potential of HyperController for efficient and stable training of reinforcement learning neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。