动态分配模型各层适配秩，用更少参数实现更高性能微调。
ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models with Heterogeneous Adaptation Needs
- 通过可学习缩放因子自动分配每头的适配秩，实现精细化控制。
- 仅用0.32%参数达到全量微调99.3%性能，多模态适配内存降41%。
- 适合需要高效微调大模型且资源受限的研究者与开发者。
传统低秩适配（LoRA）方法采用固定秩，对Transformer各层和注意力头施加统一适配，忽视其异质学习动态。本文提出自适应秩动态LoRA（ARD-LoRA），通过可学习缩放因子自动分配秩，优化目标兼顾任务性能与参数效率，引入ℓ₁稀疏性以最小化秩，及总变差正则化以保证秩变化稳定。该框架支持逐注意力头的连续、可微分秩调整。在LLAMA-3.1-70B和PaliGemma-2上的实验表明，ARD-LoRA仅需0.32%可训练参数即可达到全量微调99.3%的性能，优于DoRA和AdaLoRA等强基线，且多模态适配内存减少41%。结果确立了动态细粒度秩分配作为高效基础模型适配的关键范式。
原文摘要 · Abstract (English)
Conventional Low-Rank Adaptation (LoRA) methods employ a fixed rank, imposing uniform adaptation across transformer layers and attention heads despite their heterogeneous learning dynamics. This paper introduces Adaptive Rank Dynamic LoRA (ARD-LoRA), a novel framework that automates rank allocation through learnable scaling factors. These factors are optimized via a meta-objective balancing task performance and parameter efficiency, incorporating $\ell_1$ sparsity for minimal rank and Total Variation regularization for stable rank transitions. ARD-LoRA enables continuous, differentiable, per-head rank adaptation. Experiments on LLAMA-3.1-70B and PaliGemma-2 demonstrate ARD-LoRA's efficacy, achieving up to 99.3% of full fine-tuning performance with only 0.32% trainable parameters, outperforming strong baselines like DoRA and AdaLoRA. Furthermore, it reduces multimodal adaptation memory by 41%. These results establish dynamic, fine-grained rank allocation as a critical paradigm for efficient foundation model adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。