BoRA对权重矩阵进行纵横对称低秩适配，提升模型微调性能。
BoRA: Bi-dimensional Weight-Decomposed Low-Rank Adaptation
- 将权重矩阵的行列方向同时分解为大小与方向，实现对称优化
- 在多个基准测试中超越LoRA和DoRA，表现更优
- 适合需要高效微调大模型的研究者和工程师
近年来，参数高效微调（PEFT）方法如低秩适配（LoRA）显著提升了大规模预训练模型的适应能力。权重分解低秩适配（DoRA）通过分离权重矩阵的大小和方向成分，实现了优于LoRA的性能。然而，DoRA的改进仅局限于垂直维度，导致水平与垂直维度间存在不对称性。本文提出BoRA，作为LoRA和DoRA的创新扩展，具有纵横维度上的对称特性。我们的方法通过同时调整列方向和行方向的大小，对权重矩阵进行对称优化。大量实验表明，BoRA在多个基准测试中超越现有最优的PEFT方法，包括LoRA和DoRA，取得更优结果。
原文摘要 · Abstract (English)
In recent years, Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) have significantly enhanced the adaptability of large-scale pre-trained models. Weight-Decomposed Low-Rank Adaptation (DoRA) improves upon LoRA by separating the magnitude and direction components of the weight matrix, leading to superior performance. However, DoRA's improvements are limited to the vertical dimension, resulting in an asymmetrical pattern between horizontal and vertical dimensions. This paper introduces BoRA, an innovative extension of LoRA and DoRA, characterized by symmetrical properties across horizontal and vertical dimensions. Our approach optimizes the weight matrix symmetrically by adjusting both column-wise and row-wise magnitudes. Extensive experiments demonstrate that BoRA surpasses state-of-the-art PEFT methods, including LoRA and DoRA, achieving superior results across various benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。