arXiv:2510.08034cs.AI2025-10

针对LoRA训练慢、效果不佳问题,提出功能感知的不对称初始化方法。

AILoRA: Function-Aware Asymmetric Initialization for Low-Rank Adaptation of Large Language Models

  • 根据查询和值投影矩阵功能差异,分别注入主成分与次成分进行初始化。
  • 在多个任务上实现更快收敛,性能提升最高达2.3个点。
  • 适合需要高效微调大模型的科研与工业场景。

参数高效微调(PEFT)旨在缓解大规模预训练模型适配下游任务时的计算与内存开销。在众多PEFT方法中,低秩适应(LoRA)因表现稳健且实现简单而被广泛采用。实际应用中,LoRA通常作用于自注意力模块中的$W^Q$和$W^V$投影矩阵,实现了性能与参数效率的良好平衡。尽管LoRA已取得显著成功,但仍面临性能欠优与收敛缓慢的问题。为此,本文提出AILoRA,一种融合功能感知的非对称低秩先验的新方法。实证分析表明,$W^Q$和$W^V$在功能上存在显著差异:$W^Q$捕获任务相关的语义空间知识,对下游任务变化敏感;而$W^V$编码跨任务和层相对稳定的词元特征。基于此,AILoRA通过注入$W^Q$的主成分以保留任务适应能力,以及$W^V$的次成分以维持泛化特征表示,实现功能导向的不对称初始化。该策略使LoRA模块更充分捕捉注意力参数的分工特性,从而提升微调性能与收敛效率。

原文摘要 · Abstract (English)

Parameter-efficient finetuning (PEFT) aims to mitigate the substantial computational and memory overhead involved in adapting large-scale pretrained models to diverse downstream tasks. Among numerous PEFT strategies, Low-Rank Adaptation (LoRA) has emerged as one of the most widely adopted approaches due to its robust empirical performance and low implementation complexity. In practical deployment, LoRA is typically applied to the $W^Q$ and $W^V$ projection matrices of self-attention modules, enabling an effective trade-off between model performance and parameter efficiency. While LoRA has achieved considerable empirical success, it still encounters challenges such as suboptimal performance and slow convergence. To address these limitations, we introduce \textbf{AILoRA}, a novel parameter-efficient method that incorporates function-aware asymmetric low-rank priors. Our empirical analysis reveals that the projection matrices $W^Q$ and $W^V$ in the self-attention mechanism exhibit distinct parameter characteristics, stemming from their functional differences. Specifically, $W^Q$ captures task-specific semantic space knowledge essential for attention distributions computation, making its parameters highly sensitive to downstream task variations. In contrast, $W^V$ encodes token-level feature representations that tend to remain stable across tasks and layers. Leveraging these insights, AILoRA performs a function-aware initialization by injecting the principal components of $W^Q$ to retain task-adaptive capacity, and the minor components of $W^V$ to preserve generalizable feature representations. This asymmetric initialization strategy enables LoRA modules to better capture the specialized roles of attention parameters, thereby enhancing both finetuning performance and convergence efficiency.

LoRA微调大模型初始化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。