用低秩激活蒸馏压缩大模型，几分钟内省去100亿参数仍保持95%性能。
Lillama: Large Language Models Compression via Low-Rank Feature Distillation
- 局部蒸馏激活，用SVD初始化并联合优化师生模型激活
- 在单张A100上压缩Mixtral-8x7B，删100亿参数保留95%性能
- 适合想快速压缩大模型且不想重训的研究者和工程师
当前大语言模型结构化剪枝通常分两步：先压缩并用校准数据微调，再耗费数十亿词的持续预训练以恢复性能。这第二步必要是因为第一步会显著降低准确率。已有研究指出，预训练的Transformer权重并非天然低秩，而其激活是，这或可解释性能下降。基于此，我们提出Lillama，一种通过低秩权重局部蒸馏激活的压缩方法。利用SVD初始化，并采用结合教师与学生激活的联合损失函数，实现加速收敛与内存降低。Lillama可在单张A100 GPU上于数分钟内压缩Mixtral-8x7B，移除100亿参数后仍保持超过95%原始性能。Phi-2 3B可通过仅1300万校准令牌压缩40%，得到性能媲美同类模型的小模型。该方法还适用于非Transformer架构，使Mamba-3B压缩20%的同时维持99%性能。
原文摘要 · Abstract (English)
Current LLM structured pruning methods typically involve two steps: (1) compression with calibration data and (2) costly continued pretraining on billions of tokens to recover lost performance. This second step is necessary as the first significantly impacts model accuracy. Prior research suggests pretrained Transformer weights aren't inherently low-rank, unlike their activations, which may explain this drop. Based on this observation, we propose Lillama, a compression method that locally distills activations with low-rank weights. Using SVD for initialization and a joint loss combining teacher and student activations, we accelerate convergence and reduce memory use with local gradient updates. Lillama compresses Mixtral-8x7B within minutes on a single A100 GPU, removing 10 billion parameters while retaining over 95% of its original performance. Phi-2 3B can be compressed by 40% with just 13 million calibration tokens, resulting in a small model that competes with recent models of similar size. The method generalizes well to non-transformer architectures, compressing Mamba-3B by 20% while maintaining 99% performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。