动态调整模型各层低秩参数规模，提升微调效率与性能
ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuning
- 根据梯度重要性动态增减各层低秩秩值
- 在多个基准上优于现有方法，且节省参数量
- 适合资源受限场景，提供可解释的自适应机制
低秩适配（LoRA）已成为高效微调大规模预训练模型的常用方法，仅需极少参数更新。然而，现有方法采用固定秩值，或仅支持秩剪枝或扩展，无法在训练过程中动态匹配不同层的重要性。本文提出ElaLoRA，一种可动态剪枝和扩展秩值的自适应低秩适配框架，基于梯度重要性评分实现。据我们所知，ElaLoRA是首个在微调过程中同时支持秩剪枝与扩展的方法。多基准实验表明，其在不同参数预算下均优于现有参数高效微调（PEFT）方法。研究还验证了获得更高秩分配的层对模型性能贡献更大，为自适应策略提供了理论依据。通过引入合理且可适应的秩分配机制，ElaLoRA为资源受限环境提供了可扩展、高效的微调方案。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has become a widely adopted technique for fine-tuning large-scale pre-trained models with minimal parameter updates. However, existing methods rely on fixed ranks or focus solely on either rank pruning or expansion, failing to adapt ranks dynamically to match the importance of different layers during training. In this work, we propose ElaLoRA, an adaptive low-rank adaptation framework that dynamically prunes and expands ranks based on gradient-derived importance scores. To the best of our knowledge, ElaLoRA is the first method that enables both rank pruning and expansion during fine-tuning. Experiments across multiple benchmarks demonstrate that ElaLoRA consistently outperforms existing PEFT methods across different parameter budgets. Furthermore, our studies validate that layers receiving higher rank allocations contribute more significantly to model performance, providing theoretical justification for our adaptive strategy. By introducing a principled and adaptive rank allocation mechanism, ElaLoRA offers a scalable and efficient fine-tuning solution, particularly suited for resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。