提出洛拉微调中出现遗忘维度的谱定律,可精准预测何时发生灾难性遗忘。
The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning
- 基于权重矩阵谱特性推导出每层临界更新强度公式
- 在18个适配器、9840次层扫描中准确定位82%的遗忘阈值
- 适用于多种模型结构,可指导低遗忘微调策略
LoRA微调可能产生入侵维度:新出现的主奇异向量几乎与预训练奇异向量正交,引发灾难性遗忘。尽管已发现此类现象,但尚无理论能逐层预测其出现时机。本文通过矩形尖峰变形变换,仅依据原权重矩阵 $W$ 的谱数据,推导出每层临界更新强度 $s^ullet = \barθ/(γσ_1(BA))$,并给出更新后谱的精确特征方程,无需参数拟合。在涵盖四个密集Transformer族、一个状态空间模型、一个专家混合模型和一个编码器-解码器结构的预设研究中(18个适配器,9,840层扫描),该定律在82%的层上将实测阈值定位在两倍以内;在部署时以0.89均值AUC区分含侵入与无侵入层;在六个第三方适配器上保持一致;可预测WikiText-2困惑度开始下降的位置;两项预设评估组合达98%,外部适配器外袋验证为0.997。全微调的更新始终远低于各层阈值,解决了LoRA与全微调之间的不对称性。归一化干预证实,跨阈值层而非更新幅度是遗忘的关键,基于阈值的尖峰预算规则仅需一次SVD且无需验证轮次,使最脆弱模型遗忘减少62%而任务性能不变。
原文摘要 · Abstract (English)
LoRA fine-tuning can create intruder dimensions: new leading singular vectors of the updated weight matrix $W+BA$ that are nearly orthogonal to all pretrained singular vectors and that drive catastrophic forgetting. Since their discovery, no theory has predicted, layer by layer on measured spectra, when they appear. We derive a per-layer critical update strength $s^\ast=\barθ/(γσ_1(BA))$, computed from the measured spectrum of $W$ alone through the rectangular spiked-deformation transform, together with an exact secular-equation characterization of the updated spectrum, with no fitted parameters. In a pre-specified study spanning four dense Transformer families, a state-space model, a mixture-of-experts model, and an encoder-decoder (18 adapters, 9{,}840 layer scans), the law localizes the empirical threshold within a factor of two on $82\%$ of layers, separates intruder-bearing from intruder-free layers at deployment with a mean AUC of $0.89$, holds unchanged on six third-party adapters, and predicts where WikiText-2 perplexity begins to degrade; a combination of the two pre-specified edge evaluations reaches $98\%$ and is confirmed out-of-bag on the external adapters ($0.997$). Full fine-tuning disperses its update far below the threshold of every layer, which resolves the asymmetry between LoRA and full fine-tuning. Norm-matched interventions confirm that threshold-crossing layers, rather than update magnitude, carry the forgetting, and a spike-budget rule derived from the thresholds, requiring one SVD and no validation sweeps, reduces forgetting by $62\%$ on the most fragile model at no task cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。