arXiv:2604.08749cs.LGcs.NE2026-04

用随机固定主干+低秩适配器,仅训练少量参数就能达到全模型效果。

A Little Rank Goes a Long Way: Random Scaffolds with LoRA Adapters Are All You Need

  • 主干网络随机初始化并冻结,只训练低秩适配器。
  • 在9个基准上性能达全模型的96%-100%,参数量仅需0.5%-40%。
  • 适配器秩可估计任务内在维度,适合高效模型部署。

神经网络中真正承载任务信息的参数有多少?我们通过LottaLoRA训练范式探究此问题:所有主干权重均随机初始化并冻结,仅训练低秩LoRA适配器。在涵盖从单层分类器到900M参数Transformer的九个基准上,基于随机冻结主干的低秩适配器恢复了96%-100%的全模型性能,而训练参数仅占0.5%-40%。任务信号占据的子空间远小于参数总量所暗示的规模。三个机制性发现支撑该结果:(1)冻结主干被主动利用,学习到的缩放因子β始终为正;若主干不稳定,优化器会忽略它,所有任务信息由LoRA吸收;(2)主干可互换,任意随机初始化均可,只要保持固定;(3)性能饱和时的最小LoRA秩估计任务内在维度,类似主成分分析保留的分量数。该结构在形式上类似于沿前馈网络深度轴展开的蓄水池计算。由于主干仅由随机种子决定,模型可作为适配器加种子分发,存储开销随任务复杂度增长,而非模型大小,因此随着架构扩大,存储与内存节省持续累积。

原文摘要 · Abstract (English)

How many of a neural network's parameters actually encode task-specific information? We investigate this question with LottaLoRA, a training paradigm in which every backbone weight is drawn at random and frozen; only low-rank LoRA adapters are trained. Across nine benchmarks spanning diverse architecture families from single-layer classifiers to 900M parameter Transformers low-rank adapters over frozen random backbones recover 96-100% of fully trained performance while training only 0.5-40% of the parameters. The task-specific signal therefore occupies a subspace orders of magnitude smaller than the full parameter count suggests. Three mechanistic findings underpin this result:(1) the frozen backbone is actively exploited when static the learned scaling~$β$ remains strictly positive across all architectures but when the scaffold is destabilized, the optimizer silences it and the LoRA factors absorb all task information; (2) the frozen backbone is preferable but interchangeable any random initialization works equally well, provided it remains fixed throughout training; and (3) the minimum LoRA rank at which performance saturates estimates the intrinsic dimensionality of the task, reminiscent of the number of components retained in Principal Component Analysis (PCA). The construction is formally analogous to Reservoir Computing unfolded along the depth axis of a feedforward network. Because the backbone is determined by a random seed alone, models can be distributed as adapters plus seed a footprint that grows with task complexity, not model size, so that storage and memory savings compound as architectures scale.

LoRA参数效率随机主干适配器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。