通过保留90%~95%能量的主子空间，大幅降低大模型微调内存占用。
CERSA: Cumulative Energy-Retaining Subspace Adaptation for Memory-Efficient Fine-Tuning
- 基于SVD提取权重变化的主要成分，仅保留关键子空间
- 在图像、文本生成等任务中优于现有方法且内存更低
- 适合资源受限场景下的高效模型微调
为缓解大模型微调带来的内存压力，现有参数高效微调（PEFT）方法如LoRA依赖低秩更新，但无法充分捕捉全参数微调中的秩特性，导致性能差距。此外，这类方法仍需存储完整冻结权重，限制了其在资源受限环境下的效率。为此，本文提出累积能量保留子空间适应（CERSA），利用奇异值分解（SVD）仅保留贡献90%至95%谱能量的主成分。通过在该主子空间的低秩表示上进行微调，显著降低内存消耗。我们在多种规模和领域的模型上进行了广泛评估，涵盖图像识别、文生图生成和自然语言理解任务。实验表明，CERSA在保持高性能的同时，显著优于当前最先进的PEFT方法，且内存需求更低。代码将公开发布。
原文摘要 · Abstract (English)
To mitigate the memory constraints associated with fine-tuning large pre-trained models, existing parameter-efficient fine-tuning (PEFT) methods, such as LoRA, rely on low-rank updates. However, such updates fail to fully capture the rank characteristics of the weight modifications observed in full-parameter fine-tuning, resulting in a performance gap. Furthermore, LoRA and other existing PEFT methods still require substantial memory to store the full set of frozen weights, limiting their efficiency in resource-constrained settings. To addres these limitations, we introduce Cumulative Energy-Retaining Subspace Adaptation (CERSA), a novel fine-tuning paradigm that leverages singular value decomposition (SVD) to retain only the principal components responsible for 90% to 95% of the spectral energy. By fine-tuning low-rank representations derived from this principal subspace, CERSA significantly reduces memory consumption. We conduct extensive evaluations of CERSA across models of varying scales and domains, including image recognition, text-to-image generation, and natural language understanding. Empirical results demonstrate that CERSA consistently outperforms state-of-the-art PEFT methods while achieving substantially lower memory requirements. The code will be publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。