arXiv:2505.24254cs.LG2025-05中稿 · ICLR被引 3

提出渐进式神经坍缩框架,缓解持续学习中的灾难性遗忘问题。

Rethinking Continual Learning with Progressive Neural Collapse

  • 通过逐步扩展类原型几何结构,实现任务间最优分离
  • 在CIFAR-100上达到86.7%准确率,优于现有方法
  • 适合需要高效、灵活持续学习的工业场景

持续学习(CL)旨在构建能连续学习一系列任务的智能体,但因不同任务间知识干扰导致的灾难性遗忘仍是核心挑战。深度神经网络训练中会收敛至一种称为神经坍缩的终态,此时所有类别原型构成几何上等角紧框架(ETF),即最大且均匀分离的单纯形结构,为缓解知识干扰提供了理想目标。尽管近期研究尝试固定全局ETF,但存在不可行性和性能瓶颈。为此,本文提出渐进式神经坍缩(ProNC)框架,完全摒弃固定全局ETF,通过有原则地逐步添加新任务的类原型顶点,实现所有已知类间的最大可分性,同时最小化对旧框架的扰动。进一步将ProNC融入常用CL算法设计,并结合知识蒸馏平衡旧类目标偏移与新类对齐需求。大量实验表明,该方法显著超越基线,兼具优越的灵活性、简洁性与效率。代码已开源。

原文摘要 · Abstract (English)

Continual Learning (CL) seeks to build an agent that can continuously learn a sequence of tasks, where a key challenge, namely Catastrophic Forgetting, persists due to the potential knowledge interference among different tasks. On the other hand, deep neural networks (DNNs) are shown to converge to a terminal state termed Neural Collapse during training, where all class prototypes geometrically form a static simplex equiangular tight frame (ETF). These maximally and equally separated class prototypes make the ETF an ideal target for model learning in CL to mitigate knowledge interference. Thus inspired, several studies have emerged very recently to leverage a fixed global ETF in CL, which however suffers from key drawbacks, such as impracticability and limited performance.To address these challenges and fully unlock the potential of ETF in CL, we propose Progressive Neural Collapse (ProNC), a novel framework that completely removes the need of a fixed global ETF in CL. Specifically, ProNC progressively expands the ETF target in a principled way by adding new class prototypes as vertices for new tasks, ensuring maximal separability across all encountered classes with minimal shifts from the previous ETF. We next develop a new CL framework by plugging ProNC into commonly used CL algorithm designs, where distillation is further leveraged to balance between target shifting for old classes and target aligning for new classes. Extensive experiments show that our approach significantly outperforms related baselines while maintaining superior flexibility, simplicity, and efficiency. Our code is available at https://github.com/Continue-Edge-AI-Lab/ProNC

持续学习神经坍缩分类器设计知识蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。