遗忘不是知识消失,而是无法访问已存储的信息。
Catastrophic Forgetting as Accessibility Collapse: A Three-Level Framework for Knowledge Persistence in Continual Learning
- 将知识分为存储、表征和可访问性三层,分析遗忘机制
- 旧任务准确率从54.8%降至0%,但表征保留76%信息
- 重训分类器即可恢复75.7%性能,适合持续学习研究者
灾难性遗忘通常被理解为顺序学习中先前知识的不可逆抹除。本文提出新视角:遗忘可能并非源于任务表征的完全破坏,而是信息可访问性的丧失。我们构建一个三层次框架,区分知识的存储、表征与可访问性,并在使用ResNet-18的连续CIFAR-100分类任务上进行系列实验验证。通过检查点持久性、线性探测、表示几何、分类器重置恢复及逐层可恢复性分析发现,早期任务行为遗忘严重,准确率由54.8%跌至0%,但线性探测性能仍保持约76%原始表征信息。仅重训练最终分类器即可恢复75.7%原任务性能,无需修改主干网络。层分析显示,浅层与中间层虽在后期严重退化,仍保留高度可恢复的任务信息。投影能量与主角度分析表明,保留的知识以分布式高维表征形式存在,而非依赖单一主导子空间。结果表明,灾难性遗忘更应被定义为可访问性失效,而非表征彻底擦除,大量任务相关知识在功能遗忘后仍嵌入神经表征中。
原文摘要 · Abstract (English)
Catastrophic forgetting is commonly interpreted as the irreversible erasure of previously acquired knowledge during sequential learning. In this work, we investigate an alternative perspective: that forgetting may arise not from complete destruction of task representations but from a loss of accessibility to preserved information. We introduce a three-level framework separating knowledge storage, representation, and accessibility, and evaluate each component through a series of continual-learning experiments on sequential CIFAR-100 classification using ResNet-18. Our analysis combines checkpoint persistence, linear probing, representation geometry, classifier-reset recovery, and layer-wise recoverability experiments. We observe complete behavioral forgetting of earlier tasks, with task accuracy collapsing from 54.8% to 0%, while linear probe performance retains approximately 76% of the original representational information. Furthermore, retraining only the final classifier restores 75.7% of the original task performance without modifying the backbone network. Layer-wise analysis reveals that early and intermediate layers preserve highly recoverable task information despite severe degradation at later stages. Projection-energy and principal-angle analyses indicate that retained knowledge persists as distributed high-dimensional representations rather than through preservation of a small dominant subspace. These findings suggest that catastrophic forgetting is better characterized as an accessibility failure than complete representational erasure, and that substantial task-relevant information remains embedded within neural representations even after functional forgetting has occurred.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。