用预训练权重的稳定秩指导低秩适配,提升小数据和领域差异大场景下的模型表现。
Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes
- 基于权重矩阵的稳定秩,动态分配各层适配秩,无需额外搜索。
- 在显著领域差异的小样本任务中,性能优于现有自适应低秩方法。
- 无需复杂优化,适合资源受限环境下的高效迁移学习。
低秩适配(LoRA)在降低计算开销的同时保持了与全微调相当的性能,但在存在较大领域差异的场景下,其固定低秩结构限制了适应能力,往往需要更高秩来捕捉领域特异性复杂性。现有自适应LoRA方法虽尝试通过动态扩展或选择性分配秩来改进,但常依赖迭代剪枝、秩搜索或额外正则化等高成本技术。为此,我们提出稳定秩引导的低秩适配(SR-LoRA),利用预训练权重矩阵的稳定秩作为层间秩分配的自然先验。稳定秩反映了权重的内在维度,使SR-LoRA能够高效、合理地重新分配各层秩,增强适应性且不增加搜索开销。在具有显著领域差距的小样本任务上,实验表明SR-LoRA持续优于近期自适应LoRA变体,实现了性能与效率更优的平衡。代码已开源:https://github.com/EndoluminalSurgicalVision-IMR/SR-LoRA。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has proven effective in reducing computational costs while maintaining performance comparable to fully fine-tuned foundation models across various tasks. However, its fixed low-rank structure restricts its adaptability in scenarios with substantial domain gaps, where higher ranks are often required to capture domain-specific complexities. Current adaptive LoRA methods attempt to overcome this limitation by dynamically expanding or selectively allocating ranks, but these approaches frequently depend on computationally intensive techniques such as iterative pruning, rank searches, or additional regularization. To address these challenges, we introduce Stable Rank-Guided Low-Rank Adaptation (SR-LoRA), a novel framework that utilizes the stable rank of pre-trained weight matrices as a natural prior for layer-wise rank allocation. By leveraging the stable rank, which reflects the intrinsic dimensionality of the weights, SR-LoRA enables a principled and efficient redistribution of ranks across layers, enhancing adaptability without incurring additional search costs. Empirical evaluations on few-shot tasks with significant domain gaps show that SR-LoRA consistently outperforms recent adaptive LoRA variants, achieving a superior trade-off between performance and efficiency. Our code is available at https://github.com/EndoluminalSurgicalVision-IMR/SR-LoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。