提出矩阵乘法式微调方法LoRMA，提升大模型适配效率。
LoRMA: Low-Rank Multiplicative Adaptation for LLMs
- 用矩阵乘法替代加法更新，拓展微调表达能力。
- 在多个任务上优于传统LoRA，保持高精度与低计算开销。
- 适合资源受限场景下的大模型快速适配。
大型语言模型在自然语言处理领域展现出卓越能力，其有效性主要源于对下游任务的适应能力。然而，全量微调通常计算成本高昂。为缓解此问题，许多高效方法被提出，其中代表性方法是低秩适配（LoRA）。但LoRA及其变体采用重参数化的加性更新。本文提出低秩乘法适配（LoRMA），将加性更新范式转向更丰富的矩阵乘法变换空间。通过有效重排操作并引入秩膨胀策略，解决了矩阵乘法中的计算复杂度与秩瓶颈问题。我们在多个任务上进行了广泛实验，验证了该方法在多种评估指标下的有效性。
原文摘要 · Abstract (English)
Large Language Models have shown remarkable capabilities in the NLP domain. Their effectiveness can mainly be attributed to their ability to adapt to an array of downstream tasks. However, generally, full fine-tuning is a computationally expensive job. To mitigate this, many techniques have been developed that prime efficiency, a prominent one being Low-Rank Adaptation (LoRA). However, LoRA and its variants employ re-parametrized additive updates. In this paper, we propose Low-Rank Multiplicative Adaptation (LoRMA), which shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. We tackle challenges such as computational complexity and rank bottleneck of matrix multiplication by effectively re-ordering operations and introducing rank inflation strategies. We conduct extensive experiments to demonstrate the effectiveness of our approach in terms of various evaluation metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。