用低分离秩核优化参数高效微调，性能提升且参数减半。
LSR-Adapt: Ultra-Efficient Parameter Tuning with Matrix Low Separation Rank Kernel Adaptation
- 基于矩阵低分离秩结构设计新型核函数，压缩适配器参数。
- 相比传统方法参数减半，精度反而更高，达到新最优。
- 适合资源受限场景下的大模型轻量化微调，支持高效并行计算。
在现代大规模预训练模型的参数高效微调（PEFT）中，对神经网络权重矩阵施加有效结构假设是主流方法。然而，随着大语言模型规模不断增长，基于低秩的适配方法面临挑战。本文提出一种基于数值分析中矩阵低分离秩（LSR）表示的核化机制，用于大型网络线性层的低秩适配器矩阵，命名为低分离秩适配（LSR-Adapt）核。该超高效核表示大幅降低适配所需参数量，在几乎减少一半参数的情况下仍实现更优性能，达到当前最佳水平。该结构假设还因克罗内克运算的高度并行性，为GPU端进一步优化提供了可能。
原文摘要 · Abstract (English)
Imposing an effective structural assumption on neural network weight matrices has been the major paradigm for designing Parameter-Efficient Fine-Tuning (PEFT) systems for adapting modern large pre-trained models to various downstream tasks. However, low rank based adaptation has become increasingly challenging due to the sheer scale of modern large language models. In this paper, we propose an effective kernelization to further reduce the number of parameters required for adaptation tasks. Specifically, from the classical idea in numerical analysis regarding matrix Low-Separation-Rank (LSR) representations, we develop a kernel using this representation for the low rank adapter matrices of the linear layers from large networks, named the Low Separation Rank Adaptation (LSR-Adapt) kernel. With the ultra-efficient kernel representation of the low rank adapter matrices, we manage to achieve state-of-the-art performance with even higher accuracy with almost half the number of parameters as compared to conventional low rank based methods. This structural assumption also opens the door to further GPU-side optimizations due to the highly parallelizable nature of Kronecker computations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。