用统计检验方法自动分配LoRA的秩,提升微调效率与性能。
Statistical Inference for Rank Allocation in Low-Rank Adaptation

- 将秩分配问题转为统计假设检验,基于p值决定保留或剪枝组件。
- 在DeBERTa、BART、Qwen等模型上表现优于或相当于现有方法。
- 理论支持充分,适用于AdamW等主流优化器,适合高效微调研究者。
低秩适应(LoRA)已成为大型语言模型参数高效微调的常用方法。由于不同模块和层对下游任务贡献不均,如何在固定参数预算下合理分配秩资源,是平衡效率、表达力与泛化能力的关键问题。现有自适应秩方法主要依赖梯度敏感性和不确定性测度构建重要性评分,缺乏明确的统计解释。本文将LoRA秩分配建模为统计假设检验问题,提出StatLoRA——一种基于统计推断的秩分配方法。StatLoRA为每个LoRA组件关联一个检验统计量,并利用估计的p值在给定秩预算下决定保留或剪枝。该检验过程基于我们对随机优化轨迹的中心极限定理,证明了包括AdamW在内的多种常用优化器的渐近正态性,并推导出组件得分的渐近分布。我们在DeBERTaV3-base、BART-Large和Qwen2.5-7B模型上评估StatLoRA在自然语言理解、生成和问答任务中的表现。实验表明,在匹配秩预算下,StatLoRA性能可媲美或优于原始LoRA、AdaLoRA和IGU-LoRA。敏感性分析与实证诊断进一步验证了该检验规则的稳定性,并提供了组件得分渐近理论的经验证据。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models. Since different modules and layers may contribute unequally to downstream adaptation, allocating rank resources under a fixed parameter budget is an important problem for balancing efficiency, expressiveness, and generalization. Existing adaptive rank methods address this problem mainly through carefully designed importance scores constructed from gradient-derived sensitivity and uncertainty measures, without an explicit statistical interpretation. In this paper, we formulate LoRA rank allocation as a statistical hypothesis testing problem and propose StatLoRA, a statistical inference-based rank allocation method. StatLoRA associates each LoRA component with a test statistic and uses estimated p-values to determine which components should be retained or pruned under a prescribed rank budget. The proposed testing procedure is supported by our central limit theory for stochastic optimizer trajectories. In particular, we establish asymptotic normality for a broad class of commonly used optimizers in deep learning, including AdamW, and derive the corresponding asymptotic distributions for the proposed component scores used in hypothesis testing. We evaluate StatLoRA on LoRA fine-tuning of DeBERTaV3-base, BART-Large, and Qwen2.5-7B across natural language understanding, natural language generation, and question answering tasks. Experiments show that StatLoRA achieves comparable or better performance than vanilla LoRA, AdaLoRA, and IGU-LoRA under matched rank budgets. Sensitivity analyses and empirical diagnostics further support the stability of the proposed hypothesis-testing-based allocation rule and provide empirical evidence for the asymptotic theory of component scores.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。