无需训练即可搜索高效大模型结构，加速推理并节省显存。
Search for Efficient Large Language Models
- 提出无训练架构搜索框架，自动发现最优子网络。
- 在多个基准上超越现有最先进方法，性能更优。
- 适合需要快速部署、资源受限的大模型应用。
大语言模型（LLMs）在人工智能研究中占据主导地位。为减少内存占用和加速推理，已有多种高效技术被采用，如权重剪枝、量化和蒸馏，这些技术揭示了LLM中的冗余性。然而，多数压缩方法仅关注权重优化，忽视了最优架构的探索。传统架构搜索方法因参数量庞大而难以在LLMs上有效应用。本文提出一种无训练的架构搜索框架，用于识别保留原始模型核心能力的同时实现推理加速的最优子网络。此外，在生成继承原始模型特定权重的子网络后，我们引入一种重构算法，利用被舍弃的权重，通过少量校准数据对继承权重进行修正。相比现有最先进的无训练结构化剪枝方法，本方法在生成更小网络的同时展现出更优性能。所生成的子网络可直接降低GPU内存使用并实现推理加速。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research. Numerous efficient techniques, including weight pruning, quantization, and distillation, have been embraced to compress LLMs, targeting memory reduction and inference acceleration, which underscore the redundancy in LLMs. However, most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures. Besides, traditional architecture search methods, limited by the elevated complexity with extensive parameters, struggle to demonstrate their effectiveness on LLMs. In this paper, we propose a training-free architecture search framework to identify optimal subnets that preserve the fundamental strengths of the original LLMs while achieving inference acceleration. Furthermore, after generating subnets that inherit specific weights from the original LLMs, we introduce a reformation algorithm that utilizes the omitted weights to rectify the inherited weights with a small amount of calibration data. Compared with SOTA training-free structured pruning works that can generate smaller networks, our method demonstrates superior performance across standard benchmarks. Furthermore, our generated subnets can directly reduce the usage of GPU memory and achieve inference acceleration. Code: https://github.com/shawnricecake/search-llm
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。