让适配器的秩可学习，提升参数高效微调效果
Parameter-Efficient Fine-Tuning with Learnable Rank
- 适配器秩在训练中动态学习，不再固定
- 多任务测试中性能优于现有方法，尤其在语言理解上
- 适合追求高效微调且希望自动调整复杂度的研究者
低秩适配（LoRA）是一种流行的参数高效微调（PEFT）方法，通过将权重更新限制在低秩适配器中，引入固定的低秩归纳偏置。本文质疑固定秩约束是否最优，提出可学习秩的LoRA（LR-LoRA），允许优化器在训练过程中为每个适配层确定合适的秩。实验发现各层学习到的秩存在显著差异，变压器模型中的注意力层与MLP层表现出系统性不同的秩偏好。在多个语言理解与常识推理基准测试中，LR-LoRA在多数设置下达到领先性能，持续优于强基线，证明可学习秩提供了比固定秩更灵活有效的归纳偏置。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bias by optimizing in a low-dimensional subspace. In this work, we question whether a fixed-rank constraint is the most effective inductive bias for parameter-efficient fine-tuning. We introduce *Learnable Rank LoRA (LR-LoRA)*, a PEFT method in which the adapter rank is learned during the training process. Instead of prescribing a uniform rank for all adapter layers, LR-LoRA allows the optimizer to determine the appropriate rank for each layer. Using this approach, we find substantial layer-wise variation in the learned ranks, with the attention and MLP layers in the transformer models exhibiting systematically different rank preferences. Across a range of language understanding and commonsense reasoning benchmarks, LR-LoRA achieves state-of-the-art performance in most settings and consistently outperforms strong PEFT baselines, demonstrating that a learnable rank provides a more flexible and effective inductive bias than fixed-rank adaptations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。