用图论中心性方法剪枝大模型，大幅降低计算开销且保持高精度。
LLM-Rank: A Graph Theoretical Approach to Pruning Large Language Models
- 构建多层感知机的加权有向无环图，用改进的PageRank算节点重要性
- MLPRank平均比基线高6.09%准确率保留，LLMRank高13.42%
- 适合需高效部署大模型的研究者与工程师
大语言模型能力不断提升的同时，模型规模与部署成本也持续增长，亟需高效的推理优化技术。本文提出一种基于图论中心性度量的新型剪枝方法，有效降低模型计算需求与内存占用。具体而言，我们构建了多层感知机的加权有向无环图表示，并采用改进的加权PageRank中心性算法计算节点重要性得分。结合均匀剪枝策略，实现结构化稀疏。该方法称为MLPRank；进一步扩展至仅解码器的Transformer模型，称为LLMRank。在两种变体上均表现出色：MLPRank平均比三种主流基线高出6.09%的准确率保留，而LLMRank相比两种主流基线提升13.42%。代码已公开于https://github.com/amazon-science/llm-rank-pruning。
原文摘要 · Abstract (English)
The evolving capabilities of large language models are accompanied by growing sizes and deployment costs, necessitating effective inference optimisation techniques. We propose a novel pruning method utilising centrality measures from graph theory, reducing both the computational requirements and the memory footprint of these models. Specifically, we devise a method for creating a weighted directed acyclical graph representation of multilayer perceptrons to which we apply a modified version of the weighted PageRank centrality measure to compute node importance scores. In combination with uniform pruning this leads to structured sparsity. We call this pruning method MLPRank. Furthermore we introduce an extension to decoder-only transformer models and call it LLMRank. For both variants we demonstrate a strong performance. With MLPRank on average leading to 6.09 % higher accuracy retention than three popular baselines and 13.42 % with LLMRank compared to two popular baselines. Code is available at https://github.com/amazon-science/llm-rank-pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。