LoRA初始化方法可统一为连续谱系,性能随任务变化而优化。
Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

- 提出统一的预条件梯度初始化框架ULoRA,涵盖多种现有方法
- 在GLUE任务上达到全量微调效果,且无需额外搜索代价
- 适合追求高性能、低资源消耗的模型微调实践者
低秩适配(LoRA)以极低成本微调大模型,但其性能高度依赖适配器初始化方式。近期方法从下游损失梯度出发进行初始化:部分投影梯度至主要方向,另一些则先用损失曲率估计进行白化。本文揭示这些看似不同的方法实为同一连续统一体的特例——一个由谱白化指数和类Adam对角指数共同控制的双参数预条件梯度初始化族,称为统一LoRA(ULoRA)。在全学习率搜索下,我们发现不存在单一最优预条件强度,最佳点常位于该家族内部,而非已发表的端点。作为该家族上限的调优版ULoRA,在五项GLUE任务上使用RoBERTa-base达到或超过全量微调表现;在GSM8K任务上使用LLaMA-2-7B时,性能媲美最强基线。其可部署版本ULoRA-Auto通过测量谱统计自动选择每层指数,无需额外搜索即可逼近上限,是当前部署型LoRA方法中的顶尖水平。结果表明,LoRA初始化与曲率预条件应视为可调维度,而非固定设计。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized. Recent schemes initialize the adapters from the downstream loss gradient: some project the raw gradient onto its top directions, while others first whiten it with an estimate of the loss curvature. We show that these seemingly distinct methods are points on a single continuum: a two-parameter family of preconditioned gradient initializations, which we call Unified LoRA (ULoRA), governed by a spectral whitening exponent and an Adam-like diagonal exponent. Sweeping this family under a full learning-rate search, we find that no single fixed preconditioning strength dominates: the best operating point is task-dependent and frequently lies strictly inside the family, away from the published endpoints. Treated as an upper bound of this family, a tuned ULoRA configuration matches or exceeds full fine-tuning on all five GLUE tasks with RoBERTa-base and is competitive with the strongest baselines on GSM8K with LLaMA-2-7B. Our deployable, search-free variant, ULoRA-Auto, selects per-layer exponents from measured spectral statistics, approaches this upper bound at no additional search cost, and ranks at or near the top among deployable LoRA methods. Our results show that a principled design space for LoRA initialization and curvature preconditioning should be treated as a tunable dimension rather than a fixed design decision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。