提出MiSS结构，让LoRA更快收敛且更省资源。
MiSS: Revisiting the Trade-off in LoRA with an Efficient Shard-Sharing Structure
- 用共享矩阵更新权重块，减少训练参数
- 理论和实验证明优化复杂度降低，性能不降
- 适合需要高效微调的大模型应用
低秩适配（LoRA）是参数高效微调的常用方法，但其收敛速度慢的问题催生了众多变体。然而，现有方法难以同时提升性能、降低内存占用和计算效率。本文重新分析了LoRA收敛慢的原因，提出矩阵分片共享（MiSS）机制：使用一个初始化为零的共享可训练矩阵 $\boldsymbol{D}$ 更新原始权重矩阵的分片。为进一步保障计算效率、低内存占用和可扩展部署，引入改进版MiSS$^e$。理论分析与实验结果表明，该方法在不损失性能的前提下显著降低优化复杂度，实现性能、内存与效率之间的更优权衡。我们还对多种PEFT方法进行了全面对比，评估其内存消耗、初始化开销和计算效率，并通过绘制帕累托前沿证明MiSS处于优势位置，有效融合了前序方法的优点。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is a widely adopted technique for parameter-efficient fine-tuning, but its slow convergence has spurred the development of numerous variants. Nevertheless, existing methods often fail to improve performance, memory footprint, and computational efficiency simultaneously. To address this challenge, we revisit the causes of LoRA's slow convergence. Building on these insights, we propose Matrix Shard Sharing (MiSS), which updates shards of the original weight matrix using a single shared trainable matrix $\boldsymbol{D}$, initialized to zeros. To simultaneously ensure computational efficiency, low memory footprint, and scalable serving, we introduce MiSS$^e$. Both theoretical analysis and empirical results demonstrate that our method reduces optimization complexity without compromising performance, thereby achieving a more favorable trade-off among performance, memory, and efficiency. Furthermore, we conduct a comprehensive comparative analysis of various PEFT methods, evaluating their memory usage, initialization overhead, and computational efficiency. By mapping the Pareto frontier across these dimensions, we show that MiSS occupies a favorable position, effectively capturing the advantages of prior approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。