用切比雪夫多项式提升DQN的函数逼近能力,显著改善强化学习性能。
Beyond ReLU: Chebyshev-DQN for Enhanced Deep Q-Networks
- 在DQN中引入切比雪夫多项式基函数构建特征表示
- 在CartPole-v1上实现39%的性能提升(N=4)
- 多项式阶数需适度,过高会损害学习效果
Deep Q-Networks (DQN) 的性能高度依赖其神经网络对动作价值函数的精确逼近能力。标准函数近似器如多层感知机,在处理强化学习中的复杂价值结构时可能效率不足。本文提出一种新架构——切比雪夫-DQN(Ch-DQN),将切比雪夫多项式基嵌入DQN框架,以获得更优的特征表示。利用切比雪夫多项式的强大函数逼近能力,我们假设Ch-DQN可实现更高效的训练与更高性能。在CartPole-v1基准测试中,与参数量相当的标准DQN对比,中等阶数(N=4)的Ch-DQN实现了约39%的性能提升。然而,多项式阶数是关键超参数:高阶(N=8)反而损害学习效果。该工作验证了正交多项式基在深度强化学习中的潜力,同时揭示了模型复杂度的权衡问题。
原文摘要 · Abstract (English)
The performance of Deep Q-Networks (DQN) is critically dependent on the ability of its underlying neural network to accurately approximate the action-value function. Standard function approximators, such as multi-layer perceptrons, may struggle to efficiently represent the complex value landscapes inherent in many reinforcement learning problems. This paper introduces a novel architecture, the Chebyshev-DQN (Ch-DQN), which integrates a Chebyshev polynomial basis into the DQN framework to create a more effective feature representation. By leveraging the powerful function approximation properties of Chebyshev polynomials, we hypothesize that the Ch-DQN can learn more efficiently and achieve higher performance. We evaluate our proposed model on the CartPole-v1 benchmark and compare it against a standard DQN with a comparable number of parameters. Our results demonstrate that the Ch-DQN with a moderate polynomial degree (N=4) achieves significantly better asymptotic performance, outperforming the baseline by approximately 39\%. However, we also find that the choice of polynomial degree is a critical hyperparameter, as a high degree (N=8) can be detrimental to learning. This work validates the potential of using orthogonal polynomial bases in deep reinforcement learning while also highlighting the trade-offs involved in model complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。