让神经网络自动调节遗忘速度，提升持续学习能力
Learning to Forget: Continual Learning with Adaptive Weight Decay
- 按参数动态调整权重衰减，实现智能遗忘
- 在流式分类任务中性能超越固定衰减率方法
- 适合需要长期学习新知识的在线系统
具有有限容量的持续学习代理必须在获取新知识与保留旧知识之间取得平衡。这需要有控制地遗忘不再需要的信息，以释放容量用于新学习。权重衰减可视为一种遗忘机制，通过逐步丢弃存储在权重中的信息来实现。然而，固定的标量权重衰减会均匀地随时间推移和所有参数进行遗忘，即使某些参数编码了稳定知识，而另一些则跟踪快速变化的目标。我们提出了基于自适应衰减的遗忘（FADE），通过近似元梯度下降在线调整每个参数的权重衰减率。我们推导了在线线性设置下的FADE，并将其应用于神经网络的最后一层。实证分析表明，FADE能自动发现不同参数的差异性衰减速率，与步长自适应互补，并在在线追踪和流式分类问题上持续优于固定权重衰减。
原文摘要 · Abstract (English)
Continual learning agents with finite capacity must balance acquiring new knowledge with retaining the old. This requires controlled forgetting of knowledge that is no longer needed, freeing up capacity to learn. Weight decay, viewed as a mechanism for forgetting, can serve this role by gradually discarding information stored in the weights. However, a fixed scalar weight decay drives this forgetting uniformly over time and uniformly across all parameters, even when some encode stable knowledge while others track rapidly changing targets. We introduce Forgetting through Adaptive Decay (FADE), which adapts per-parameter weight decay rates online via approximate meta-gradient descent. We derive FADE for the online linear setting and apply it to the final layer of neural networks. Our empirical analysis shows that FADE automatically discovers distinct decay rates for different parameters, complements step-size adaptation, and consistently improves over fixed weight decay across online tracking and streaming classification problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。