用非线性函数生成低秩适配器，大幅减少参数量
Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions
- 用径向基函数生成低秩矩阵的基向量，替代显式存储
- 相同参数量下有效秩更高，微调效果更优
- 适合资源受限场景下的高效模型适配
低秩适配（LoRA）通过两个低秩矩阵的乘积近似预训练权重矩阵的更新。然而，标准LoRA采用显式秩范式，增加模型容量需添加更多行或列（即基向量），导致参数量显著增长。本文发现这些基向量存在显著冗余，可由轻量非线性函数紧凑表示。为此提出生成式低秩适配器（GenLoRA），将显式基向量存储替换为非线性基向量生成。具体而言，GenLoRA为每个低秩矩阵维护一个隐向量，并使用一组轻量径向基函数（RBFs）合成基向量。每个RBF所需参数远少于显式基向量，从而实现更高参数效率。跨多个数据集和架构的大量实验表明，GenLoRA在更小参数预算下达到更高有效LoRA秩，带来更优微调性能。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) approximates the update of a pretrained weight matrix using the product of two low-rank matrices. However, standard LoRA follows an explicit-rank paradigm, where increasing model capacity requires adding more rows or columns (i.e., basis vectors) to the low-rank matrices, leading to substantial parameter growth. In this paper, we find that these basis vectors exhibit significant parameter redundancy and can be compactly represented by lightweight nonlinear functions. Therefore, we propose Generative Low-Rank Adapter (GenLoRA), which replaces explicit basis vector storage with nonlinear basis vector generation. Specifically, GenLoRA maintains a latent vector for each low-rank matrix and employs a set of lightweight radial basis functions (RBFs) to synthesize the basis vectors. Each RBF requires far fewer parameters than an explicit basis vector, enabling higher parameter efficiency in GenLoRA. Extensive experiments across multiple datasets and architectures show that GenLoRA attains higher effective LoRA ranks under smaller parameter budgets, resulting in superior fine-tuning performance. The code is available at https://anonymous.4open.science/r/GenLoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。