arXiv:2602.06043cs.LGcs.AI2026-02被引 1

用共享低秩子空间实现高效持续学习,避免遗忘且节省大量参数。

Shared LoRA Subspaces for almost Strict Continual Learning

  • 构建单一共享低秩子空间,动态融合新任务知识。
  • 相比传统LoRA,参数减少100倍,内存节省281倍。
  • 适合需要长期更新、资源受限的AI系统部署。

将大模型高效且持续地适配到新任务对实际应用至关重要,但面临灾难性遗忘和重训练成本高的挑战。尽管低秩适应(LoRA)等参数高效微调方法降低了计算开销,却缺乏严格持续学习和知识整合机制,且不依赖数据回放或多个适配器。本文提出Share,一种新的参数高效持续微调方法:通过学习并动态更新单一共享低秩子空间,实现跨多任务和模态的无缝适应。Share构建基础子空间以提取过往任务的核心知识,并通过识别关键子空间方向增量集成新信息。每个新任务的知识被融入此不断演化的子空间中,促进正向知识迁移,同时最小化灾难性干扰。该方法在参数上实现高达100倍减少,在内存上节省281倍,性能接近联合训练模型。单个Share模型可替代数百个特定任务的LoRA适配器,支持可扩展、异步的持续学习。在图像分类、自然语言理解、3D姿态估计和文本生成图像等多个任务上的实验验证了其有效性,使Share成为大规模AI系统中实用且可扩展的终身学习解决方案。

原文摘要 · Abstract (English)

Adapting large pretrained models to new tasks efficiently and continually is crucial for real-world deployment but remains challenging due to catastrophic forgetting and the high cost of retraining. While parameter-efficient tuning methods like low rank adaptation (LoRA) reduce computational demands, they lack mechanisms for strict continual learning and knowledge integration, without relying on data replay, or multiple adapters. We propose Share, a novel approach to parameter efficient continual finetuning that learns and dynamically updates a single, shared low-rank subspace, enabling seamless adaptation across multiple tasks and modalities. Share constructs a foundational subspace that extracts core knowledge from past tasks and incrementally integrates new information by identifying essential subspace directions. Knowledge from each new task is incorporated into this evolving subspace, facilitating forward knowledge transfer, while minimizing catastrophic interference. This approach achieves up to 100x parameter reduction and 281x memory savings over traditional LoRA methods, maintaining performance comparable to jointly trained models. A single Share model can replace hundreds of task-specific LoRA adapters, supporting scalable, asynchronous continual learning. Experiments across image classification, natural language understanding, 3D pose estimation, and text-to-image generation validate its effectiveness, making Share a practical and scalable solution for lifelong learning in large-scale AI systems.

持续学习参数高效LoRA知识融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。