发现持续学习中适配器有低秩冗余，可复用旧适配器提升效率
When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning
- 通过分析发现不同任务的适配器空间重叠度高，可复用
- 新方法减少20%-70%活跃适配器数，性能持平或超越现有方法
- 适合追求高效持续学习的模型部署场景
低秩适配（LoRA）已成为高效微调大模型的标准工具。在持续学习（CL）中，传统假设是每个新任务需独立适配器。本文通过实证和结构分析挑战该假设：不同任务的LoRA适配器存在显著低秩冗余——由不同任务训练的适配器所张成的子空间高度重叠，许多情况下早期适配器能准确表示后续任务。基于此，我们提出LiteLoRA，一种即插即用的门控机制，在训练时学习是否启用新适配器或复用已有低秩表示。该方法在标准CL基准上将活跃适配器数量减少20%-70%，同时匹配或超越当前最佳性能，揭示了结构冗余普遍存在，且选择性学习足以实现稳定性与可塑性的平衡。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant low-rank redundancy: the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by 20-70% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。