提出AID方法，用分段丢弃激活来防止神经网络失去适应能力。
Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
- 在每个预激活区间应用不同概率的丢弃，生成子网络。
- 在CIFAR、TinyImageNet等数据集上保持持续学习中的适应能力。
- 适合需要长期适应新任务的场景，如持续学习和强化学习。
神经网络训练中的塑性损失是关键挑战，限制模型对新任务或数据分布变化的适应能力。本文提出AID（Activation by Interval-wise Dropout），一种受丢弃法启发的新方法，通过在每个预激活区间以不同概率应用丢弃来生成子网络。理论分析表明，AID能对网络进行正则化，促进类似于无塑性损失的深层线性网络的行为。我们在多个基准测试中验证了AID的有效性，包括在标准图像分类数据集CIFAR10、CIFAR100和TinyImageNet上的持续学习任务。此外，我们还证明AID在雅达利学习环境（Arcade Learning Environment）基准中提升了强化学习性能。
原文摘要 · Abstract (English)
Plasticity loss, a critical challenge in neural network training, limits a model's ability to adapt to new tasks or shifts in data distribution. This paper introduces AID (Activation by Interval-wise Dropout), a novel method inspired by Dropout, designed to address plasticity loss. Unlike Dropout, AID generates subnetworks by applying Dropout with different probabilities on each preactivation interval. Theoretical analysis reveals that AID regularizes the network, promoting behavior analogous to that of deep linear networks, which do not suffer from plasticity loss. We validate the effectiveness of AID in maintaining plasticity across various benchmarks, including continual learning tasks on standard image classification datasets such as CIFAR10, CIFAR100, and TinyImageNet. Furthermore, we show that AID enhances reinforcement learning performance in the Arcade Learning Environment benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。