用恒定状态的Mamba模型实现持续学习,不存旧数据也能高效适应新任务。
Learning Mamba as a Continual Learner: Meta-Learning Selective State Space Models for Continual Learning
- 用Mamba构建恒定大小状态的持续学习器,避免存储历史数据
- 在多个场景下表现优于传统方法,泛化能力更强
- 适合需要低内存、快速适应的新任务学习场景
持续学习(CL)从非平稳数据流中学习,无需存储或重新训练所有已见样本。元持续学习(MCL)将CL建模为序列预测,并元学习持续学习者本身作为序列模型,常用Transformer。但其需线性增长的键值缓存存储所有过去表示,违背了不存储历史数据的目标。而具有恒定状态的序列模型(如线性注意力模型和状态空间模型,SSMs)天然满足此要求;然而,因过往样本被压缩并混合于有限状态中,这类学习者更难进行元学习,早期基于核的方法在MCL上表现不佳。为此,我们提出适用于MCL的可选择性状态空间模型,采用实际中的Mamba作为主干模型。通过引入选择性正则化进行元训练,监督压缩状态内的隐式关联,并适用于各类恒定状态模型。进一步开展系统性实证研究,分析Mamba及其他恒定状态模型在多种MCL场景下的表现,涵盖有效性、泛化性与模型设计。结果表明该方法具备优异性能与强泛化能力,证明恒定状态学习者在高效持续学习与适应中的潜力。
原文摘要 · Abstract (English)
Continual learning (CL) learns from a non-stationary data stream without storing or re-training on all seen samples. Meta-continual learning (MCL) casts CL as sequence prediction and meta-learns the continual learner itself as a sequence model, with Transformers as natural choices. However, despite decent performance, a Transformer learner relies on a linearly growing key-value cache to store all past representations, conflicting with CL's objective of not storing all seen samples. Sequence models with a constant-size state, e.g., linear-attention models and state-space models (SSMs), match CL's requirement by definition; however, with past samples compressed and mixed in the bounded state, such learners are harder to meta-learn, and earlier kernel-based ones performed poorly on MCL. We thus aim to obtain effective continual learners with constant-size states and make them work for MCL. By formulating the selective SSM for MCL, we propose MambaCL, with Mamba as the main model in practice. We meta-train the learner with a proposed selectivity regularization, which supervises the implicit associations within the compressed state and applies across constant-state models. Furthermore, we conduct a systematic empirical study of how Mamba and other constant-state models behave across various MCL scenarios, covering effectiveness, generalization, and model designs. The results highlight promising performance and strong generalization, demonstrating the potential of constant-state learners for efficient continual learning and adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。