TaRA让LoRA初始化更准，提升微调效果
TaRA: Training-Aware Low-Rank Adaptation Initialization
- 根据全模型梯度设计低秩矩阵初始值
- 训练初期梯度逼近度显著提升，性能更优
- 适合追求高效微调的AI研究者与工程师
低秩适应（LoRA）已成为参数高效微调（PEFT）的主流方法，但其性能对初始化高度敏感，源于低秩分解带来的信息瓶颈。现有方法通过利用预训练权重、激活或梯度的主成分来构建高质量的LoRA初始值，但未直接考虑全模型的训练动态。本文提出训练感知的低秩适应初始化（TaRA），使低秩因子引起的梯度尽可能逼近对应全秩权重矩阵的梯度。基于数学推导，TaRA在训练初期显著提升梯度保真度，且计算开销可忽略。在多种复杂微调任务中，TaRA持续优于现有最先进方法，提供一种简单、稳健且可扩展的有效LoRA初始化方案。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has become a de facto standard for parameter-efficient fine-tuning (PEFT), yet its performance is highly sensitive to initialization due to the information bottleneck imposed by low-rank decomposition. Existing approaches attempt to construct high-quality LoRA initializations by exploiting principal components of pretrained weights, activations, or gradients. However, these methods do not directly account for the training dynamics of the full-rank model. In this paper, we propose Training-aware Low-Rank Adaptation Initialization (TaRA), a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix. Derived from a mathematical formulation, TaRA improves gradient fidelity at the start of training while introducing negligible computational overhead. Across diverse and challenging fine-tuning tasks, TaRA consistently outperforms prior state-of-the-art methods, establishing a simple, robust, and scalable solution for effective LoRA initialization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。