用5%的低秩模块实现高效微调，避免模型遗忘与性能下降
Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation
- 通过指示函数选择性激活低秩适配块，减少参数更新
- 仅需5%活跃模块即可保持性能，计算开销大幅降低
- 适合资源有限但需保留模型泛化能力的场景
将深度学习模型适应新领域通常需要昂贵的重新训练，并存在灾难性遗忘风险。虽然微调可实现领域特定适配，但可能削弱对分布外（OOD）数据的鲁棒性，影响泛化性能。预训练零样本模型如CLIP虽具强泛化能力，但在微调后可能出现鲁棒性下降。基于任务自适应参数共享（TAPS），我们提出一种简单高效的参数高效微调（PEFT）方法，利用指示函数选择性激活低秩适配（LoRA）模块。该方法最小化知识损失，保持域偏移下的泛化能力，且相比传统微调显著降低计算成本。实验表明，仅需5%的活跃模块即可实现有效微调，大幅提升效率。在CLIP和DINO-ViT等预训练模型上的评估验证了本方法的广泛适用性与有效性，能有效维持性能与知识保留。
原文摘要 · Abstract (English)
Adapting deep learning models to new domains often requires computationally intensive retraining and risks catastrophic forgetting. While fine-tuning enables domain-specific adaptation, it can reduce robustness to distribution shifts, impacting out-of-distribution (OOD) performance. Pre-trained zero-shot models like CLIP offer strong generalization but may suffer degraded robustness after fine-tuning. Building on Task Adaptive Parameter Sharing (TAPS), we propose a simple yet effective extension as a parameter-efficient fine-tuning (PEFT) method, using an indicator function to selectively activate Low-Rank Adaptation (LoRA) blocks. Our approach minimizes knowledge loss, retains its generalization strengths under domain shifts, and significantly reduces computational costs compared to traditional fine-tuning. We demonstrate that effective fine-tuning can be achieved with as few as 5\% of active blocks, substantially improving efficiency. Evaluations on pre-trained models such as CLIP and DINO-ViT demonstrate our method's broad applicability and effectiveness in maintaining performance and knowledge retention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。