C-LoRA让大模型持续学习更高效，不增加参数也能避免遗忘。
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Models
- 用可学习的路由矩阵动态分配任务更新，替代多适配器
- 在多个任务上达到当前最佳准确率，参数量几乎不变
- 适合需要长期更新的AI系统，如智能助手、机器人
低秩适应（LoRA）是一种高效的微调方法，广泛应用于自然语言处理和计算机视觉。现有LoRA方法在静态环境下表现优异，但在动态学习中因依赖多个适配模块而效率下降，增加推理开销。我们提出持续低秩适应（C-LoRA），一种面向持续学习的LoRA新扩展。C-LoRA通过可学习的路由矩阵动态管理跨任务的参数更新，实现已学子空间的高效重用，并通过正交性约束最小化干扰与遗忘。与需为每项任务配置独立适配器的方法不同，C-LoRA实现了任务适应的统一架构，在序列学习场景中兼顾可扩展性与参数效率。C-LoRA在基准测试中取得最优准确率与参数效率，并提供了关于路由矩阵在知识保留与迁移中作用的理论分析，构建了一个可扩展的持续学习框架。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that has been extensively applied in areas such as natural language processing and computer vision. Existing LoRA fine-tuning approaches excel in static environments but struggle in dynamic learning due to reliance on multiple adapter modules, increasing overhead and complicating inference. We propose Continual Low-Rank Adaptation (C-LoRA), a novel extension of LoRA for continual learning. C-LoRA uses a learnable routing matrix to dynamically manage parameter updates across tasks, ensuring efficient reuse of learned subspaces while enforcing orthogonality to minimize interference and forgetting. Unlike existing approaches that require separate adapters for each task, C-LoRA enables a integrated approach for task adaptation, achieving both scalability and parameter efficiency in sequential learning scenarios. C-LoRA achieves state-of-the-art accuracy and parameter efficiency on benchmarks while providing theoretical insights into its routing matrix's role in retaining and transferring knowledge, establishing a scalable framework for continual learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。