1LoRA用单向量压缩微调，参数少、内存省、效果更好。
1LoRA: Summation Compression for Very Low-Rank Adaptation
- 用特征求和固定压缩，单向量可训练解压，极简参数设计。
- 每层参数比现有方法减少90%以上，内存和计算成本显著降低。
- 适合资源受限场景，尤其适合全层均衡微调的模型优化。
参数高效微调（PEFT）方法通过调整远少于原始模型矩阵的参数，改变了大模型在下游任务上的微调方式。本文研究‘极低秩’情形，即对每个线性层使用最少数量的可调参数。提出1LoRA（求和低秩适配），一种在计算、参数和内存上均高效的微调方法，采用特征求和作为固定压缩，以单一可训练向量作为解压机制。与当前主流的LoRA、VeRA及最新MoRA等方法相比，1LoRA每层参数更少，显著降低内存占用和计算开销。我们在多个微调任务中广泛评估该方法，结果表明其不仅性能更优，且在参数、内存和计算效率上全面领先。此外，由于内存效率高，1LoRA可更均匀地跨层微调，而非仅聚焦特定层（如注意力层），进一步提升性能。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) methods have transformed the approach to fine-tuning large models for downstream tasks by enabling the adjustment of significantly fewer parameters than those in the original model matrices. In this work, we study the "very low rank regime", where we fine-tune the lowest amount of parameters per linear layer for each considered PEFT method. We propose 1LoRA (Summation Low-Rank Adaptation), a compute, parameter and memory efficient fine-tuning method which uses the feature sum as fixed compression and a single trainable vector as decompression. Differently from state-of-the-art PEFT methods like LoRA, VeRA, and the recent MoRA, 1LoRA uses fewer parameters per layer, reducing the memory footprint and the computational cost. We extensively evaluate our method against state-of-the-art PEFT methods on multiple fine-tuning tasks, and show that our method not only outperforms them, but is also more parameter, memory and computationally efficient. Moreover, thanks to its memory efficiency, 1LoRA allows to fine-tune more evenly across layers, instead of focusing on specific ones (e.g. attention layers), improving performance further.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。