通过约束层间关系漂移,缓解增量学习中的灾难性遗忘。
SR$^2$-LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning

- 分析层间表示关系漂移是遗忘主因,提出新修复机制。
- 在标准基准上显著降低遗忘率,任务越多效果越明显。
- 适合需要持续学习、避免旧知识丢失的场景。
预训练模型结合参数高效微调(PEFT)在类别增量学习(CIL)中展现出潜力,但适应新任务时仍存在灾难性遗忘问题。本文通过分析层间关系漂移(inter-layer relation drift),即新任务学习过程中层间表示关系的渐进性破坏,揭示了遗忘根源。理论上证明,该漂移会减小已有任务的分类边界,从而降低整体性能。为此,提出自修复层间关系低秩适配(SR²-LoRA),通过构建当前与历史模型在当前任务样本上的关系矩阵,并对齐其奇异值来抑制漂移。理论表明,奇异值对齐比逐元素对齐更具鲁棒性。大量实验表明,SR²-LoRA能有效缓解遗忘,在任务数量增加时优势更显著。代码已开源。
原文摘要 · Abstract (English)
Pre-trained models with parameter-efficient fine-tuning (PEFT) have demonstrated promising potential for class-incremental learning (CIL), yet catastrophic forgetting still persists when adapting models to new tasks. In this paper, we present a novel perspective on catastrophic forgetting through the analysis of inter-layer relation drift, i.e., the progressive disruption of relationships among layer-wise representations during the learning of new tasks. We theoretically show that the increase of such drift reduces the classification margins of previously learned tasks, thereby degrading overall model performance. To address this issue, we propose \underline{S}elf-\underline{R}ectifying inter-layer \underline{R}elation Low-Rank Adaptation~(SR$^2$-LoRA), a simple yet effective method that mitigates catastrophic forgetting by constraining inter-layer relation drift. Specifically, SR$^2$-LoRA constructs the relation matrices induced by the previous and current models on current-task samples, and aligns the corresponding singular values. We further theoretically show that this alignment exhibits greater robustness to estimation perturbations than direct entry-wise alignment. Extensive experiments on standard CIL benchmarks demonstrate that SR$^2$-LoRA effectively mitigates catastrophic forgetting, with its advantages becoming more pronounced as the number of tasks increases. Code is available in the \href{https://github.com/FqWan24/SR-2-LoRA}{repository}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。