同时微调与压缩大模型，提升小模型性能。
TuneComp: Joint Fine-tuning and Compression for Large Foundation Models
- 边微调边通过剪枝+低秩结构蒸馏压缩模型
- 比先微调后压缩的性能提升显著
- 适合资源受限下部署高性能大模型
为减少后训练阶段的模型规模，通常在微调后采用知识蒸馏、低秩近似和剪枝等压缩方法。然而，顺序进行微调与压缩会损失性能，且中间需构建更大的模型。本文旨在缩小这一差距，通过直接构建一个更小的模型来指导下游任务。我们提出联合微调与压缩的方法，逐步将模型蒸馏至剪枝后的低秩结构。实验表明，该方法显著优于其他顺序压缩方式。
原文摘要 · Abstract (English)
To reduce model size during post-training, compression methods, including knowledge distillation, low-rank approximation, and pruning, are often applied after fine-tuning the model. However, sequential fine-tuning and compression sacrifices performance, while creating a larger than necessary model as an intermediate step. In this work, we aim to reduce this gap, by directly constructing a smaller model while guided by the downstream task. We propose to jointly fine-tune and compress the model by gradually distilling it to a pruned low-rank structure. Experiments demonstrate that joint fine-tuning and compression significantly outperforms other sequential compression methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。