Saten通过稀疏增强张量网络，实现大模型后训练阶段的高效压缩。
Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models
- 引入稀疏增强张量网络，提升低秩张量化LLM在微调中的表现
- 全模型压缩下仍保持最优精度与压缩效率
- 适合资源受限设备上部署大型语言模型
大型语言模型（LLM）在资源受限设备上的高效部署至关重要。低秩张量压缩技术（如张量列车，TT）被广泛研究用于过度参数化的神经网络。然而，由于预训练LLM具有高秩特性且无法获取预训练数据，该技术在下游任务的后训练压缩中仍面临挑战。本文研究了微调过程中张量化LLM的表现，提出稀疏增强张量网络（Saten）以提升性能。Saten框架支持全模型压缩，实验表明其在张量化语言模型中显著提升了准确率与压缩效率，达到当前最优水平。
原文摘要 · Abstract (English)
The efficient implementation of large language models (LLMs) is crucial for deployment on resource-constrained devices. Low-rank tensor compression techniques, such as tensor-train (TT) networks, have been widely studied for over-parameterized neural networks. However, their applications to compress pre-trained large language models (LLMs) for downstream tasks (post-training) remains challenging due to the high-rank nature of pre-trained LLMs and the lack of access to pretraining data. In this study, we investigate low-rank tensorized LLMs during fine-tuning and propose sparse augmented tensor networks (Saten) to enhance their performance. The proposed Saten framework enables full model compression. Experimental results demonstrate that Saten enhances both accuracy and compression efficiency in tensorized language models, achieving state-of-the-art performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。