给离线强化学习的策略网络加正则化，平均提升6%性能。
The Role of Deep Learning Regularizations on Actors in Offline RL
- 在离线强化学习中对策略网络使用标准正则化方法。
- 在三个连续动作的D4RL环境中平均性能提升6%。
- 适合关注离线强化学习泛化能力的研究者。
深度学习正则化技术（如丢弃、层归一化、权重衰减）广泛用于现代神经网络，通常能提升训练鲁棒性和泛化能力。然而，在强化学习领域，这些技术应用有限，通常仅用于价值函数估计器，且可能产生负面效果。这一问题在离线强化学习中尤为突出，尽管其更接近监督学习，但研究关注度较低。近期研究表明，虽然批评者网络可构建得足够强大，但策略网络的泛化仍是瓶颈。本研究通过实验表明，在离线强化学习的演员-评论家算法中，对策略网络应用标准正则化技术，可在两个算法和三个不同连续动作的D4RL领域上实现平均6%的性能提升。
原文摘要 · Abstract (English)
Deep learning regularization techniques, such as dropout, layer normalization, or weight decay, are widely adopted in the construction of modern artificial neural networks, often resulting in more robust training processes and improved generalization capabilities. However, in the domain of Reinforcement Learning (RL), the application of these techniques has been limited, usually applied to value function estimators (Hiraoka et al., 2021; Smith et al., 2022), and may result in detrimental effects. This issue is even more pronounced in offline RL settings, which bear greater similarity to supervised learning but have received less attention. Recent work in continuous offline RL (Park et al., 2024) has demonstrated that while we can build sufficiently powerful critic networks, the generalization of actor networks remains a bottleneck. In this study, we empirically show that applying standard regularization techniques to actor networks in offline RL actor-critic algorithms yields improvements of 6% on average across two algorithms and three different continuous D4RL domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。