提出无需预训练的在线学习方法,突破现有方法对数据先验的依赖。
Are Current Continual Learning Methods Truly Agnostic? Introducing OPRE, a Step Toward Agnostic Continual Learning
- 设计在线数据压缩算法,通过输入空间两个明确标准丢弃冗余信息。
- 在CIFAR-10/100上性能媲美顶尖在线持续学习方法,无需预训练特征提取器。
- 仅做最小可解释假设,适合追求模型通用性与可解释性的研究者。
为实现持续学习(CL),必须克服灾难性遗忘问题。当前评估方法通常将已知同质数据集拆分为多个任务顺序学习。我们指出,多数CL方法隐含引入了对后续数据的先验信息,无法真正实现无偏。以依赖预训练特征提取器的方法为例,其会降低模型对新数据的泛化能力;其他方法也存在类似先验偏差。为此,我们提出在线补丁冗余消除器(OPRE),一种基于输入空间两个显式标准进行信息丢弃的在线数据压缩算法。使用随机初始化分类器时,OPRE在CIFAR-10和CIFAR-100上的表现达到当前最优在线持续学习方法水平,且无需预训练特征提取器,同时在相同内存预算下优于GDumb。结果从信息论角度为持续学习提供了新的实证视角。
原文摘要 · Abstract (English)
In order to achieve Continual Learning (CL), the problem of catastrophic forgetting, one that has plagued neural networks since their inception, must be overcome. The evaluation of continual learning methods relies on splitting a known homogeneous dataset and learning the associated tasks one after the other. We argue that most CL methods introduce a priori information about the data to come and cannot be considered agnostic. We exemplify this point with the case of methods relying on pretrained feature extractors, which are still used in CL. After showing that pretrained feature extractors imply a loss of generality with respect to the data that can be learned by the model, we then discuss other kinds of a priori information introduced in other CL methods. We then present the Online Patch Redundancy Eliminator (OPRE), an online dataset-compression algorithm that discards information through two explicit, input-space criteria. With a classifier that was randomly initialized at test time, OPRE's performance matches reported state-of-the-art online continual-learning methods on CIFAR 10 and CIFAR-100 without any pretrained feature extractor, and outperforms GDumb at an identical memory budget-while making only minimal and interpretable assumptions about the data to come. We frame these results as an empirical, information-theoretic perspective on continual learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。