通过重缩放基础模型提升微调效果，增强模型泛化能力
$α$-LoRA: Effective Fine-Tuning via Base Model Rescaling
- 引入α-LoRA，通过重缩放基础模型权重提升微调性能
- 理论证明在高维二分类中可显著改善泛化能力
- 适用于大模型微调，尤其适合资源受限场景
微调已被证明在少量数据下有效适配预训练模型以完成新任务。现有主流方法为重参数化技术，通过在冻结权重矩阵上叠加可训练矩阵来更新目标模块，其中最具代表性的是低秩适应（LoRA），近年来受到广泛关注。本文提出一类新型重参数化方法α-LoRA，旨在提升微调模型的泛化能力。我们利用随机矩阵理论，在高维二分类设置下建立了该方法的有效性，并通过更贴近实际的实验（如大语言模型微调）进一步验证了理论结果。
原文摘要 · Abstract (English)
Fine-tuning has proven to be highly effective in adapting pre-trained models to perform better on new desired tasks with minimal data samples. Among the most widely used approaches are reparameterization methods, which update a target module by augmenting its frozen weight matrix with an additional trainable weight matrix. The most prominent example is Low Rank Adaption (LoRA), which gained significant attention in recent years. In this paper, we introduce a new class of reparameterization methods for transfer learning, designed to enhance the generalization ability of fine-tuned models. We establish the effectiveness of our approach in a high-dimensional binary classification setting using tools from Random Matrix Theory, and further validate our theoretical findings through more realistic experiments, such as fine-tuning LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。