低秩微调中谱不平衡导致遗忘，新方法通过解耦优化缓解此问题。
Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation
- 解耦更新幅度与方向，在限制流形上进行约束优化
- 在多个视觉语言任务上显著减少前向与后向遗忘
- 适用于高效持续学习场景，尤其适合资源受限部署
参数高效持续学习旨在不遗忘先前知识的前提下，将预训练模型适配到顺序任务。现有方法多关注避免过去更新的干扰，却忽视当前任务更新如何自然保留已有知识。从知识分解视角看，低秩适配存在严重的奇异值谱不平衡：少数主导成分吸收大部分更新能量，从而更易破坏已有知识，且更易受后续任务干扰。为此，我们解耦任务更新的幅值与方向结构，将其建模为在受限Stiefel流形上的约束优化问题，并采用与主流视觉-语言模型优化器兼容的投影一阶方法求解。该方法有效缓解了前后向遗忘，在多个基准上持续优于基线方法。代码已开源：https://github.com/haodotgu/EBLoRA。
原文摘要 · Abstract (English)
Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge. Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge. From a knowledge-decomposition perspective, we observe that low-rank adaptations exhibit highly imbalanced singular value spectra: a few dominant components absorb most of the adaptation energy, thereby (i) more likely to disrupt previously acquired knowledge and (ii) making the update more vulnerable to interference from subsequent tasks. To enable explicit balance among components, we decouple the magnitude of the task update from its directional structure and formulate it as a constrained optimization problem on a restricted Stiefel manifold. We address this problem using a projected first-order method compatible with standard deep-learning optimizers used in vision-language models. Our method mitigates both backward and forward forgetting, consistently outperforming continual learning baselines. The implementation code is available at https://github.com/haodotgu/EBLoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。