根据层特性动态分配低秩适配器规模，提升微调效率
LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
- 基于训练时轻量级费舍尔估计，动态分配各层适配器秩
- 在GLUE和MathInstruct上优于或媲美现有方法，参数更少
- 无需搜索，适合资源受限场景的高效微调
低秩适配（LoRA）广泛用于参数高效微调，但现有方法通常对所有Transformer层分配相同适配器秩，忽略了各层不同的适配需求。本文从理论和实验两方面证明，均匀秩分配本质上次优。为此提出LAARA（Layer-Aware Adaptive Rank Allocation）框架，一种无需搜索的自适应方法，利用训练中计算的轻量级对角费舍尔估计动态分配秩。LAARA结合逐投影归一化、对数压缩、混合适配器重要性估计及投票变更抑制机制，实现稳定高效的秩自适应。在GLUE和MathInstruct基准上的实验表明，LAARA始终达到或超越主流方法如LoRA、AdaLoRA、DyLoRA和Bitfit的性能，同时使用更少可训练参数。结果表明，费舍尔引导的秩分配为自适应参数高效微调提供了合理且有效的基础。代码已公开于：https://anonymous.4open.science/r/LAARA-D305/LAARA.py
原文摘要 · Abstract (English)
Low-Rank Adaptation is widely used for parameter-efficient fine-tuning, yet existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements. In this work, we show theoretically and empirically that uniform rank allocation is fundamentally suboptimal. Motivated by this observation, we propose LAARA (Layer Aware Adaptive Rank Allocation framework), a search-free framework that dynamically allocates ranks using lightweight diagonal Fisher estimates computed during training. LAARA combines projection-wise normalization, logarithmic compression, blended adapter importance estimation, and a vote-to-change dampening mechanism to produce stable and efficient rank adaptation. Experiments on GLUE and MathInstruct benchmark demonstrate that LAARA consistently matches or outperforms popular state of the art approaches such as LoRA, AdaLoRA, DyLoRA, and Bitfit while using significantly fewer trainable parameters. Our results show that Fisher-guided rank allocation provides a principled and effective foundation for adaptive parameter-efficient fine-tuning. The code is publicly available at: https://anonymous.4open.science/r/LAARA-D305/LAARA.py
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。