用高阶张量分解压缩大模型微调参数，省资源还保持性能。
LoRTA: Low Rank Tensor Adaptation of Large Language Models
- 采用高阶CP分解替代传统矩阵，更紧凑地表示模型更新。
- 在多个任务上参数量减少，性能与现有方法相当。
- 适合资源受限场景下高效微调大语言模型。
低秩适配（LoRA）是一种流行的参数高效微调（PEFT）方法，通过在每一层使用低秩矩阵参数化模型更新，显著降低微调时的可训练参数数量和资源开销。然而，由于依赖低秩矩阵结构，其可训练参数数量仍存在下限。近期工作尝试通过低秩张量参数化来缓解此问题，但仅利用层间冗余，或采用人工设计的张量化方案引入额外超参数。本文提出一种更高阶的Candecomp/Parafac（CP）分解，相较于现有矩阵与张量基PEFT方法，实现更紧凑且灵活的表示。在自然语言理解、指令微调、偏好优化及蛋白质折叠等多个基准上的实验表明，该方法可在保持相当性能的同时显著减少参数量。
原文摘要 · Abstract (English)
Low Rank Adaptation (LoRA) is a popular Parameter Efficient Fine Tuning (PEFT) method that effectively adapts large pre-trained models for downstream tasks. LoRA parameterizes model updates using low-rank matrices at each layer, significantly reducing the number of trainable parameters and, consequently, resource requirements during fine-tuning. However, the lower bound on the number of trainable parameters remains high due to the use of the low-rank matrix model. Recent works have addressed this limitation by proposing low rank tensor parameterizations for model updates. However, they only exploit redundancy across layers, or tensorize individual matrices using ad-hoc schemes that introduce additional hyperparameters. In this work, we propose a higher-order Candecomp/Parafac (CP) decomposition, enabling a more compact and flexible representation compared to existing matrix and tensor based PEFT methods. Our experiments on Natural Language Understanding, Instruction Tuning, Preference Optimization and Protein Folding benchmarks demonstrate that our method can achieve a reduction in the number of parameters while maintaining comparable performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。