用权重正则化提升低秩持续学习的稳定性,兼顾存储与效率。
Revisiting Weight Regularization for Low-Rank Continual Learning
- 通过EWC正则化共享低秩更新,缓解任务干扰。
- 在多个基准上表现优于现有低秩方法,平衡稳定与灵活。
- 适合需要高效持续学习的大模型应用者参考。
大规模预训练模型(PTMs)的持续学习(CL)近年受到广泛关注,研究重点从从零训练转向持续微调。这催生了参数高效持续学习(PECL)新范式,通常通过引入特定任务模块(如低秩适配器)来缓解任务干扰。然而,弹性权重巩固(EWC)等权重正则化技术在此范式中仍被忽视。本文重新审视低秩CL中的权重正则化,提出EWC-LoRA方法:通过在共享低秩空间中对参数重要性进行估计,以正则化方式减少任务干扰。该方法保持恒定的存储与推理开销,不受任务数量影响。实验表明,EWC-LoRA在多个基准上均显著优于现有低秩方法,实现了更优的稳定-可塑权衡。结果表明,在低秩参数化下,权重正则化仍是有效缓解干扰的机制。代码已开源。
原文摘要 · Abstract (English)
Continual Learning (CL) with large-scale pre-trained models (PTMs) has recently gained wide attention, shifting the focus from training from scratch to continually adapting PTMs. This has given rise to a promising paradigm: parameter-efficient continual learning (PECL), where task interference is typically mitigated by assigning a task-specific module during training, such as low-rank adapters. However, weight regularization techniques, such as Elastic Weight Consolidation (EWC)-a key strategy in CL-remain underexplored in this new paradigm. In this paper, we revisit weight regularization in low-rank CL as a new perspective for mitigating task interference in PECL. Unlike existing low-rank CL methods, we mitigate task interference by regularizing a shared low-rank update through EWC, thereby keeping the storage requirement and inference costs constant regardless of the number of tasks. Our proposed method EWC-LoRA leverages a low-rank representation to estimate parameter importance over the full-dimensional space. This design offers a practical, computational- and memory-efficient solution for CL with PTMs, and provides insights that may inform the broader application of regularization techniques within PECL. Extensive experiments on various benchmarks demonstrate the effectiveness of EWC-LoRA, achieving a stability-plasticity trade-off superior to existing low-rank CL approaches. These results indicate that, even under low-rank parameterizations, weight regularization remains an effective mechanism for mitigating task interference. Code is available at: https://github.com/yaoyz96/low-rank-cl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。