arXiv:2602.07940cs.AI2026-02被引 2

不依赖回放的持续学习新方法,提升预训练模型适应新任务能力。

MePo: Meta Post-Refinement for Rehearsal-Free General Continual Learning

  • 通过构造伪任务序列,用元学习优化预训练模型
  • 在多个数据集上实现12.56%以上性能提升
  • 适合希望少依赖记忆回放的持续学习研究者

为应对外部环境的不确定性,智能系统需持续学习复杂动态环境并实时响应,这一能力称为通用持续学习(GCL),面临在线数据流和任务边界模糊等挑战。尽管预训练模型(PTMs)推动了传统持续学习发展,但其难以有效处理单次遍历中的多样化、时间混合信息,导致性能受限。受神经科学中元可塑性与重构记忆启发,本文提出元后精炼(MePo)方法,从预训练数据构建伪任务序列,设计双层元学习机制对预训练主干进行精炼,相当于延长预训练阶段,显著加速下游GCL任务的表征学习适应。此外,引入元协方差矩阵作为预训练表示空间的参考几何结构,使GCL能利用二阶统计信息实现鲁棒输出对齐。该方法为即插即用策略,在无需回放条件下于多个GCL基准和预训练检查点上取得显著提升(如CIFAR-100、ImageNet-R、CUB-200上分别达15.10%、13.36%、12.56%),代码已开源。

原文摘要 · Abstract (English)

To cope with uncertain changes of the external world, intelligent systems must continually learn from complex, evolving environments and respond in real time. This ability, collectively known as general continual learning (GCL), encapsulates practical challenges such as online datastreams and blurry task boundaries. Although leveraging pretrained models (PTMs) has greatly advanced conventional continual learning (CL), these methods remain limited in reconciling the diverse and temporally mixed information along a single pass, resulting in sub-optimal GCL performance. Inspired by meta-plasticity and reconstructive memory in neuroscience, we introduce here an innovative approach named Meta Post-Refinement (MePo) for PTMs-based GCL. This approach constructs pseudo task sequences from pretraining data and develops a bi-level meta-learning paradigm to refine the pretrained backbone, which serves as a prolonged pretraining phase but greatly facilitates rapid adaptation of representation learning to downstream GCL tasks. MePo further initializes a meta covariance matrix as the reference geometry of pretrained representation space, enabling GCL to exploit second-order statistics for robust output alignment. MePo serves as a plug-in strategy that achieves significant performance gains across a variety of GCL benchmarks and pretrained checkpoints in a rehearsal-free manner (e.g., 15.10\%, 13.36\%, and 12.56\% on CIFAR-100, ImageNet-R, and CUB-200 under Sup-21/1K). Our source code is available at \href{https://github.com/SunGL001/MePo}{MePo}

持续学习预训练模型元学习无回放

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。