LoRA能显著减少Transformer模型在连续微调中的遗忘，让旧知识更稳定。
Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning: Controlled Empirical Evidence and Frozen-Backbone Representation Probes
- 用低秩适配（LoRA）控制参数更新范围，只微调少量新参数。
- 实验显示遗忘率从全量微调的19.9%降至0.6%，效果显著且统计显著。
- 适合关注模型持续学习、避免灾难性遗忘的研究者与工程师。
在五个完整验证的BERT-base任务序列（RTE→MRPC→CoLA→SST-2）中，全量微调导致平均遗忘率达19.9%±4.8%，而标准LoRA（r=8，仅更新query/value模块）将遗忘率降至0.6%±1.4%（配对t检验，p=0.002，Cohen's d_s=3.12）。任务级分析证实该效果非整体平均掩盖的真实提升。罗伯特基（RoBERTa-base）实验复现相同模式，最强的EWC基线仍达15.5%±1.4%遗忘。六任务扩展揭示：平均遗忘低可能掩盖任务间异质性。精细冻结消融实验表明，当冻结参数超过约95%时，遗忘明显下降，仅微调分类器或浅层适配器的方案接近LoRA表现。在GPT-2和RoBERTa上的任务相似性探针显示，冻结主干架构能更好保持跨任务特征相似性，逐步解冻削弱稳定性，全量微调在最后一层表现出最明显差异。结果支持一种机制解释：LoRA的有效性主要源于主干冻结保留了更稳定的共享特征结构。因此，标准LoRA不仅是连续编码器适配的强基准，也是研究选择性可塑性如何影响干扰的有力探针。
原文摘要 · Abstract (English)
Sequential fine-tuning of pretrained language encoders often overwrites previously acquired capabilities, but the forgetting behavior of parameter-efficient updates remains under-characterized. We present a controlled empirical study of Low-Rank Adaptation (LoRA) in sequential transformer encoder fine-tuning with companion representation probes that test a frozen-backbone explanation of its robustness. In five full-validation BERT-base reruns on an RTE->MRPC->CoLA->SST-2 sequence, full fine-tuning yields 19.9%+/-4.8% average forgetting, whereas standard LoRA (r=8, query/value modules) yields 0.6%+/-1.4% (paired t-test, p=0.002, Cohen's d_s=3.12). Task-level analyses confirm this reduction is not merely an aggregate effect. Secondary experiments on RoBERTa-base show the same pattern, and the strongest EWC baseline remains at 15.5%+/-1.4% forgetting. A six-task extension reveals that low average forgetting can hide strong task-level heterogeneity. Fine-grained freezing ablations show a marked forgetting drop once frozen parameters exceed roughly 95%, with classifier-only and shallow-adapter baselines approaching LoRA. Companion task-similarity probes in GPT-2 and RoBERTa show the same directional story: frozen-backbone regimes preserve higher inter-task similarity than full fine-tuning, gradual unfreezing weakens stability, and full fine-tuning exhibits its clearest divergence at the final transformer layer. These results support a restrained mechanistic interpretation: LoRA helps largely because backbone freezing preserves a more stable shared feature scaffold. We position standard LoRA as both a strong empirical baseline for sequential encoder adaptation and a useful probe of how selective plasticity shapes interference in transformer continual learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。