用稀疏关系编码和局部学习,减少持续学习中的灾难性遗忘。
The Art of Not Forgetting A Local Learning Architecture for Continual Learning

- 输入转为稀疏关系码,通过两级竞争记忆存储
- 在15个语言领域实验中,遗忘率显著低于Transformer基线
- 适合研究持续学习机制的算法设计者
我们提出CMP(认知记忆原语),一种持续学习架构,将输入表示为稀疏关系码,存入两级竞争记忆,并通过局部更新学习,无需对特征生成系统进行端到端反向传播。在受控的域增量字节级语言建模任务中,CMP表现出比参数匹配的Transformer+在线弹性权重巩固(EWC)低得多的后向迁移损失。在三组重复的15域实验中,CMP表现出稳定的遗忘行为;头对头对比和领域顺序分析均显示其在报告设置下持续低于基线模型的遗忘率。同时,我们报告了与Transformer相比的单领域准确率差距、视觉基准上的零结果,以及未能与独立提升精度的机制结合,体现了对正负结果的全面披露。这些发现表明,稀疏表示、局部学习与持久记忆的结合是缓解灾难性遗忘的有前景方向,也提示需进一步探究学习规则、表征和架构设计各自的作用。
原文摘要 · Abstract (English)
We introduce CMP (Cognitive Memory Primitive), a continual-learning architecture that repre?sents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns through local updates without end-to-end backpropagation through its feature-generating system. We investigate whether combining sparse representations, local learning, and persistent memory can reduce catastrophic forgetting relative to conventional backpropagation-based continual?learning approaches. On a controlled domain-incremental byte-level language modeling protocol, CMP demonstrates substantially lower backward transfer than a parameter-matched Trans?former trained with online Elastic Weight Consolidation (EWC). Across a three-seed replicated 15-domain experiment, CMP exhibits stable forgetting behavior, while separate head-to-head comparisons and domain-order analyses show consistently lower forgetting than the evaluated Transformer baseline under the reported experimental settings. We report these findings alongside a substantial single-domain accuracy gap relative to the Transformer, a null result on a vision benchmark, and a documented failure to combine CMP with an independent accuracy-improving mechanism, reflecting our commitment to reporting both positive and negative outcomes. These results suggest that the combination of sparse representations, local learning, and persistent memory is a promising direction for continual learning, while motivating further investigation into the respective roles of learning rules, representations, and architectural design in mitigating catastrophic forgetting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。