自动搜索最优子网络，压缩大模型同时提升性能与推理速度。
Compressing Large Language Models with Automated Sub-Network Search
- 将模型压缩转化为神经架构搜索问题，自动剪枝注意力头、神经元和层。
- 在11个任务上平均性能提升9.85%，设备端延迟降低22%。
- 适合需要高效部署大模型的场景，如移动端或边缘计算。
大型语言模型（LLMs）展现出卓越的推理能力，可在常识推理和指令遵循等多样化任务中实现强泛化。然而，随着模型规模增大，推理成本显著上升，生命周期内累积开销巨大。本文研究了针对LLM的模型压缩方法，在减小模型体积的同时提升下游任务表现。将该问题建模为神经架构搜索，自动剪枝注意力头、神经元和层，以寻找性能与设备端延迟之间的帕累托最优子网络集合。相比现有结构化剪枝方法及从预训练模型中提取的小型微调子网络，本方法在11项多样化下游任务上平均提升9.85%，设备端延迟最高改善22%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) demonstrate exceptional reasoning abilities, enabling strong generalization across diverse tasks such as commonsense reasoning and instruction following. However, as LLMs scale, inference costs become increasingly prohibitive, accumulating significantly over their life cycle. In this paper we consider model compression for LLMs to reduce model size while improving downstream task performance. We phrase this as a neural architecture search problem that automatically prunes structural components, such as attention heads, neurons, and layers by searching for the Pareto-optimal set of sub-networks balancing between performance and on-device latency. Compared to state-of-the-art structural pruning approaches and fine-tuned smaller sub-networks extracted from the pre-trained model, our method achieves upto 9.85% improvement on average on 11 diverse downstream tasks, while achieving up to 22% improvement of on-device latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。