Dual LoRA通过分离参数更新的幅度与方向，提升低秩微调性能。
Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
- 将低秩矩阵拆分为幅度和方向两组，分别控制更新强度和方向。
- 在多个NLP任务上优于标准LoRA及当前最优变体，相同参数量下表现更优。
- 方法简单有效，适合追求高效微调且关注精度的LLM应用者。
低秩适应（LoRA）是参数高效微调（PEFT）中广泛使用的方法，用于将预训练大语言模型（LLMs）适配到下游任务。然而，由于其低秩假设，基于LoRA训练的模型性能常不理想。本文提出一种新方法Dual LoRA，通过引入归纳偏置改进性能。具体地，将低秩矩阵分为两组：幅度组控制参数是否更新及更新程度，方向组决定参数更新方向，更贴近基于梯度优化的全量微调过程。该方法仅需在幅度组加入ReLU函数、方向组加入符号函数即可实现。我们在RoBERTa、DeBERTa及LLaMA-1/2/3等多个基线模型上进行了广泛实验，涵盖自然语言理解（NLU）和常识推理任务。结果表明，本方法在相同可训练参数数量下，持续优于标准LoRA及其最先进变体。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) is one of the most popular methods among parameter-efficient fine-tuning (PEFT) methods to adapt pre-trained large language models (LLMs) to specific downstream tasks. However, the model trained based on LoRA often has an unsatisfactory performance due to its low-rank assumption. In this paper, we propose a novel method called Dual LoRA to improve the performance by incorporating an inductive bias into the original LoRA. Specifically, we separate low-rank matrices into two groups: the magnitude group to control whether or not and how far we should update a parameter and the direction group to decide whether this parameter should move forward or backward, to better simulate the parameter updating process of the full fine-tuning based on gradient-based optimization algorithms. We show that this can be simply achieved by adding a ReLU function to the magnitude group and a sign function to the direction group. We conduct several experiments over a wide range of NLP tasks, including natural language understanding (NLU) and commonsense reasoning datasets on RoBERTa, DeBERTa, and LLaMA-1/2/3 as baseline models. The results show that we consistently outperform LoRA and its state-of-the-art variants with the same number of trainable parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。