通过降低模型输出波动，缓解持续强化学习中的适应力衰退问题。
Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn
- 从训练批次外数据的输出波动（churn）角度分析适应力下降机制。
- 减少churn可防止神经正切核秩崩溃，并自适应调整梯度步长。
- 新方法C-CHAIN在多个基准测试中超越现有基线，提升持续学习性能。
塑性，即智能体适应新任务、环境或分布的能力，对持续学习至关重要。本文从‘波动’（churn）视角研究深度持续强化学习中的塑性损失：由小批量训练引发的批次外数据输出变异性。我们证明：(1) 塑性损失伴随churn加剧，源于神经正切核（NTK）矩阵的逐步秩下降；(2) 减少churn有助于防止秩坍缩，并自适应调整常规RL梯度的步长。为此，我们提出持续波动近似缩减（C-CHAIN），并在OpenAI Gym Control、ProcGen、DeepMind Control Suite和MinAtar等多个持续学习基准上验证其有效性，结果优于现有基线方法。
原文摘要 · Abstract (English)
Plasticity, or the ability of an agent to adapt to new tasks, environments, or distributions, is crucial for continual learning. In this paper, we study the loss of plasticity in deep continual RL from the lens of churn: network output variability for out-of-batch data induced by mini-batch training. We demonstrate that (1) the loss of plasticity is accompanied by the exacerbation of churn due to the gradual rank decrease of the Neural Tangent Kernel (NTK) matrix; (2) reducing churn helps prevent rank collapse and adjusts the step size of regular RL gradients adaptively. Moreover, we introduce Continual Churn Approximated Reduction (C-CHAIN) and demonstrate it improves learning performance and outperforms baselines in a diverse range of continual learning environments on OpenAI Gym Control, ProcGen, DeepMind Control Suite, and MinAtar benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。