提出在线持续学习的表征对齐方法，缓解遗忘并加速收敛。
CLA: Latent Alignment for Online Continual Self-Supervised Learning
- 通过对比当前与历史表征实现隐空间对齐，防止模型遗忘。
- 在相同计算预算下，训练速度更快且性能超越现有方法。
- 适用于资源受限的在线持续学习场景，尤其适合预训练初期使用。
自监督学习（SSL）能构建泛化能力强的隐空间表征。然而，针对在线持续学习（Online CL）场景——数据以小批次到达、模型需满足固定计算预算且无明确任务边界——现有的SSL方法极为有限。本文提出一种新型策略：持续表征对齐（CLA），通过将当前模型学到的表征与过往表征对齐，缓解遗忘问题。实验发现，CLA可显著加快在线训练过程的收敛速度，在相同计算预算下优于现有最先进方法。更意外的是，将CLA作为预训练早期阶段的协议，相比完整的独立同分布（i.i.d.）预训练，能获得更优的最终性能。
原文摘要 · Abstract (English)
Self-supervised learning (SSL) is able to build latent representations that generalize well to unseen data. However, only a few SSL techniques exist for the online CL setting, where data arrives in small minibatches, the model must comply with a fixed computational budget, and task boundaries are absent. We introduce Continual Latent Alignment (CLA), a novel SSL strategy for Online CL that aligns the representations learned by the current model with past representations to mitigate forgetting. We found that our CLA is able to speed up the convergence of the training process in the online scenario, outperforming state-of-the-art approaches under the same computational budget. Surprisingly, we also discovered that using CLA as a pretraining protocol in the early stages of pretraining leads to a better final performance when compared to a full i.i.d. pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。