提出新型激活函数，让强化学习模型更少遗忘，提升样本与内存效率。
Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions
- 设计具稀疏输出和梯度的象形激活函数，缓解灾难性遗忘。
- 在值函数算法中替换激活函数，显著提升模型抗遗忘能力。
- 适合追求高效、低存储消耗的强化学习应用开发者。
灾难性遗忘是强化学习领域长期存在的挑战（Ring 1994, Rivest and Precup 2003）。尽管已有研究提出有效缓解方法，但多集中于算法层面。我们对神经网络架构特性如何导致遗忘尚不充分理解。本研究通过分析激活函数在训练动态中的作用，揭示了除稀疏表示外，激活函数的梯度稀疏性也对减少遗忘至关重要。基于此，我们提出一类新型激活函数——象形激活函数（elephant activation functions），能同时生成稀疏输出和稀疏梯度。实验表明，仅在基于值的算法神经网络中替换传统激活函数为象形激活函数，即可显著增强模型对灾难性遗忘的鲁棒性，从而提升强化学习的样本效率与内存效率。
原文摘要 · Abstract (English)
Catastrophic forgetting has remained a significant challenge for efficient reinforcement learning for decades (Ring 1994, Rivest and Precup 2003). While recent works have proposed effective methods to mitigate this issue, they mainly focus on the algorithmic side. Meanwhile, we do not fully understand what architectural properties of neural networks lead to catastrophic forgetting. This study aims to fill this gap by studying the role of activation functions in the training dynamics of neural networks and their impact on catastrophic forgetting in reinforcement learning setup. Our study reveals that, besides sparse representations, the gradient sparsity of activation functions also plays an important role in reducing forgetting. Based on this insight, we propose a new class of activation functions, elephant activation functions, that can generate both sparse outputs and sparse gradients. We show that by simply replacing classical activation functions with elephant activation functions in the neural networks of value-based algorithms, we can significantly improve the resilience of neural networks to catastrophic forgetting, thus making reinforcement learning more sample-efficient and memory-efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。