提出新方法提升低秩微调稳定性，防止模型遗忘和冲突。
HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation
- 将更新分解为主导与正交两通道，利用奇异值加权控制
- 在相同预算下显著降低预训练性能退化与多适配器合并失败
- 适合持续学习、知识编辑等对干扰敏感的任务场景
在资源受限条件下微调基础模型依赖参数高效微调（PEFT），LoRA 是标准的模块化方案。然而，LoRA 存在谱干扰问题：低秩更新常集中在预训练权重的主奇异方向上，扰动通用能力，导致灾难性遗忘和脆弱的多适配器合并。为此，我们提出 HiP-LoRA，一种谱感知的适应框架。利用缓存的预训练层奇异值分解（SVD），HiP-LoRA 将更新分解为两个通道：主通道位于主导奇异子空间内，残差低秩通道位于正交补空间中。主通道引入基于奇异值加权的稳定性预算，持续平衡预训练行为保持与任务特定可塑性。在 Llama-3.1-8B 上的实验表明，在匹配预算下，HiP-LoRA 显著降低预训练性能退化和多适配器合并失败率，在持续微调与知识编辑等干扰敏感任务中稳健优于基线。
原文摘要 · Abstract (English)
Adapting foundation models under resource budgets relies heavily on Parameter-Efficient Fine-Tuning (PEFT), with LoRA being a standard modular solution. However, LoRA suffers from spectral interference. Low-rank updates often concentrate energy on the leading singular directions of pretrained weights, perturbing general capabilities and causing catastrophic forgetting and fragile multi-adapter merging. To resolve this, we propose HiP-LoRA, a spectrum-aware adaptation framework. Utilizing the cached singular value decomposition (SVD) of pretrained layers, HiP-LoRA decomposes updates into two channels: a principal channel within the dominant singular subspace, and a residual low-rank channel in the orthogonal complement. A singular-value-weighted stability budget on the principal channel continuously balances pretrained behavior preservation with task-specific plasticity. Experiments on Llama-3.1-8B demonstrate that under matched budgets, HiP-LoRA drastically reduces pretraining degradation and multi-adapter MergeFail, robustly outperforming baselines in interference-sensitive tasks like continual tuning and knowledge editing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。