平衡软硬关系，让模型记住旧知识且不占内存。
Memory-efficient Continual Learning with Neural Collapse Contrastive
- 用新损失函数同时捕捉样本间的软硬关系。
- 无需记忆存储，性能媲美需缓存的主流方法。
- 适合注重隐私、无法保存数据的场景使用。
对比学习显著提升了表示质量，增强了持续学习（CL）中的知识迁移。然而灾难性遗忘仍是主要挑战，因为基于对比的方法主要关注样本间的“软关系”或“软性”，这些关系随数据分布变化而漂移，导致不同任务间表示重叠。近期发现的神经坍缩现象通过聚焦样本间的“硬关系”或“硬度”及固定原型，在CL中展现出潜力。但该方法忽视了“软性”，而这种僵化关注也可能将旧类别表示拉向当前类别，加剧遗忘。基于此，我们提出焦点神经坍缩对比（FNC^2），一种新型表示学习损失，能有效平衡软硬关系。此外，引入硬度-软度蒸馏（HSD）损失，逐步保留跨任务关系所获知识。本方法优于现有最先进方法，尤其在降低内存依赖方面表现突出。令人惊讶的是，即使不使用记忆，其性能仍可媲美基于回放的方法，为数据隐私问题提供有力解决方案。
原文摘要 · Abstract (English)
Contrastive learning has significantly improved representation quality, enhancing knowledge transfer across tasks in continual learning (CL). However, catastrophic forgetting remains a key challenge, as contrastive based methods primarily focus on "soft relationships" or "softness" between samples, which shift with changing data distributions and lead to representation overlap across tasks. Recently, the newly identified Neural Collapse phenomenon has shown promise in CL by focusing on "hard relationships" or "hardness" between samples and fixed prototypes. However, this approach overlooks "softness", crucial for capturing intra-class variability, and this rigid focus can also pull old class representations toward current ones, increasing forgetting. Building on these insights, we propose Focal Neural Collapse Contrastive (FNC^2), a novel representation learning loss that effectively balances both soft and hard relationships. Additionally, we introduce the Hardness-Softness Distillation (HSD) loss to progressively preserve the knowledge gained from these relationships across tasks. Our method outperforms state-of-the-art approaches, particularly in minimizing memory reliance. Remarkably, even without the use of memory, our approach rivals rehearsal-based methods, offering a compelling solution for data privacy concerns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。