低精度稀疏模型高效微调，资源受限环境也能用。
SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models
- 提出端到端低精度稀疏参数高效微调方法。
- 稀疏权重与低秩适配器融合不丢精度和稀疏性。
- 支持不同精度的量化权重与适配器合并，适合边缘部署。
大型预训练模型（如大语言模型）已广泛应用于各类场景，通常通过微调适配特定任务。本文提出SQFT，一种面向低精度稀疏参数高效微调的大模型端到端解决方案，可在资源受限环境下实现高效模型调整。创新性地实现了稀疏权重与低秩适配器的融合，既保持稀疏性又不损失精度，克服了以往方法的局限。同时，针对量化权重与适配器精度不一致的问题，支持在目标数值格式下无损合并，保障性能。在多种模型、适应场景及多级稀疏度下验证了SQFT的有效性。代码与模型已开源：https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning。
原文摘要 · Abstract (English)
Large pre-trained models (LPMs), such as large language models, have become ubiquitous and are employed in many applications. These models are often adapted to a desired domain or downstream task through a fine-tuning stage. This paper proposes SQFT, an end-to-end solution for low-precision sparse parameter-efficient fine-tuning of LPMs, allowing for effective model manipulation in resource-constrained environments. Additionally, an innovative strategy enables the merging of sparse weights with low-rank adapters without losing sparsity and accuracy, overcoming the limitations of previous approaches. SQFT also addresses the challenge of having quantized weights and adapters with different numerical precisions, enabling merging in the desired numerical format without sacrificing accuracy. Multiple adaptation scenarios, models, and comprehensive sparsity levels demonstrate the effectiveness of SQFT. Models and code are available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。