提出新方法让大模型结构化剪枝更准更快，保持高精度同时提升推理速度。
Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention
- 用幂变换对齐不同层剪枝分数分布，解决异构问题
- 保留剪枝方向符号信息，提升优化一致性
- 基于百分位剔除异常值，适合实际部署场景
本文提出一种改进的大型语言模型（LLM）结构化剪枝方法，解决了将非结构化剪枝技术自适应特征保留（AFR）迁移至结构化剪枝时面临的三大挑战：异构剪枝评分间的分布不匹配、优化方向一致性的符号信息丢失，以及异常值的影响。为此，我们提出统一框架，结合幂变换实现非线性分布对齐、符号保持的评分聚合，以及基于百分位的异常值剔除。在 Llama-3-8B、Vicuna-v1.5-13B 与 LLaVA-v1.5-13B 上的实验表明，该方法在保持接近非结构化剪枝精度的同时，通过结构化剪枝实现了实用的推理加速。
原文摘要 · Abstract (English)
This paper proposes an improved structured pruning method for large language models (LLMs) that addresses key challenges in adapting Adaptive Feature Retention (AFR), an unstructured pruning technique, to structured pruning. When applying AFR to structured pruning, three major problems arise: distribution mismatch between heterogeneous pruning scores, loss of sign information indicating optimization direction consistency, and influence of outliers. To address these issues, we propose a unified approach combining power transformation for nonlinear distribution alignment, sign-preserving score aggregation, and percentile-based outlier removal. Experiments on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B demonstrate that our method maintains accuracy comparable to unstructured pruning while achieving practical inference speedup through structured pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。