用正交正则化提升强化学习持续训练能力
Parseval Regularization for Continual Reinforcement Learning
- 通过保持权重矩阵正交性，缓解持续学习中的泛化退化
- 在网格世界、CARL和MetaWorld任务上显著提升新任务适应能力
- 适合研究持续强化学习与模型可训练性的研究人员
当深度神经网络在任务序列上训练时，会出现可塑性下降、可训练性丧失和先验偏见等问题，表现为难以有效学习新任务。本文提出使用Parseval正则化，通过维持权重矩阵的正交性，保留有益的优化特性，从而改善持续强化学习中的训练表现。实验在一系列网格世界、CARL和MetaWorld任务中验证了该方法的有效性。通过全面消融实验，我们识别出其优势来源，并研究了与网络可训练性相关的指标，包括权重矩阵秩、权重范数和策略熵的影响。
原文摘要 · Abstract (English)
Loss of plasticity, trainability loss, and primacy bias have been identified as issues arising when training deep neural networks on sequences of tasks -- all referring to the increased difficulty in training on new tasks. We propose to use Parseval regularization, which maintains orthogonality of weight matrices, to preserve useful optimization properties and improve training in a continual reinforcement learning setting. We show that it provides significant benefits to RL agents on a suite of gridworld, CARL and MetaWorld tasks. We conduct comprehensive ablations to identify the source of its benefits and investigate the effect of certain metrics associated to network trainability including weight matrix rank, weight norms and policy entropy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。