提出新方法构建逻辑回归的紧凑记忆，显著提升持续学习准确率。
Compact Memory for Continual Logistic Regression
- 基于海森矩阵匹配，用概率PCA估计最优记忆
- 在Split-ImageNet上达74%准确率，接近批量训练的77.6%
- 适用于小内存场景，为深层持续学习提供新思路
尽管持续学习取得进展，其性能仍不及批量训练。为避免灾难性遗忘，需构建紧凑的过去知识记忆，但即便对仅有一到两层的浅层神经网络，也尚无明确解决方案。本文提出一种针对逻辑回归的紧凑记忆构建方法，基于Khan与Swaroop（2021）的理论：存在此类模型的最优记忆。我们将其转化为海森矩阵匹配问题，并采用概率PCA进行估计。相比经验回放，本方法显著提升准确率：在Split-ImageNet上，当记忆大小仅为数据量0.3%时，准确率达60%，而回放仅30%；将记忆增至2%时，准确率提升至74%，逼近批量训练的77.6%。该工作开辟了紧凑记忆构建的新方向，未来或可拓展至持续深度学习。
原文摘要 · Abstract (English)
Despite recent progress, continual learning still does not match the performance of batch training. To avoid catastrophic forgetting, we need to build compact memory of essential past knowledge, but no clear solution has yet emerged, even for shallow neural networks with just one or two layers. In this paper, we present a new method to build compact memory for logistic regression. Our method is based on a result by Khan and Swaroop [2021] who show the existence of optimal memory for such models. We formulate the search for the optimal memory as Hessian-matching and propose a probabilistic PCA method to estimate them. Our approach can drastically improve accuracy compared to Experience Replay. For instance, on Split-ImageNet, we get 60% accuracy compared to 30% obtained by replay with memory-size equivalent to 0.3% of the data size. Increasing the memory size to 2% further boosts the accuracy to 74%, closing the gap to the batch accuracy of 77.6% on this task. Our work opens a new direction for building compact memory that can also be useful in the future for continual deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。