高记忆性样本更易被遗忘，但对性能提升关键。
What is the role of memorization in Continual Learning?
- 用记忆得分评估样本在持续学习中的遗忘倾向。
- 高记忆分样本遗忘更快，但对性能提升不可或缺。
- 大缓存时优先保留高记忆分样本效果更优。
记忆化影响深度学习模型性能。以往研究多关注泛化与隐私中的记忆现象，本文聚焦增量学习场景。实验表明，记忆得分高的样本比普通样本遗忘更快。研究发现，记忆化对达到最高性能是必要的；但在小内存情况下，遗忘普通样本更为重要。随着缓存容量增大，高记忆分样本的重要性上升。本文提出一种记忆代理，并用于缓存策略优化，证明在大缓存条件下，优先保留高记忆代理得分的样本有益于训练效果。
原文摘要 · Abstract (English)
Memorization impacts the performance of deep learning algorithms. Prior works have studied memorization primarily in the context of generalization and privacy. This work studies the memorization effect on incremental learning scenarios. Forgetting prevention and memorization seem similar. However, one should discuss their differences. We designed extensive experiments to evaluate the impact of memorization on continual learning. We clarified that learning examples with high memorization scores are forgotten faster than regular samples. Our findings also indicated that memorization is necessary to achieve the highest performance. However, at low memory regimes, forgetting regular samples is more important. We showed that the importance of a high-memorization score sample rises with an increase in the buffer size. We introduced a memorization proxy and employed it in the buffer policy problem to showcase how memorization could be used during incremental training. We demonstrated that including samples with a higher proxy memorization score is beneficial when the buffer size is large.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。