不需训练就能大幅降低大模型推理开销，同时保持高精度。
SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Models
- 基于隐藏状态聚类，无训练地剪枝无关专家参数
- 50%剪枝时比现有方法高20%以上，20%剪枝仅损失2.2%性能
- 适合追求高效部署的大模型应用开发者
大型语言模型在自然语言处理任务中表现卓越，但推理阶段的高计算成本仍是主要瓶颈。本文提出无需训练的稀疏专家激活剪枝（SEAP），通过识别任务相关的专家激活模式，选择性保留关键参数以降低推理开销。受大模型中隐藏状态与激活的聚类特性启发，SEAP在不牺牲任务性能的前提下显著提升计算效率。实验表明，50%剪枝率下，SEAP性能超越WandA和FLAP超过20%；20%剪枝时仅损失2.2%精度。结果验证了SEAP在大规模语言模型优化中的可扩展性与有效性。
原文摘要 · Abstract (English)
Large Language Models have achieved remarkable success across various natural language processing tasks, yet their high computational cost during inference remains a major bottleneck. This paper introduces Sparse Expert Activation Pruning (SEAP), a training-free pruning method that selectively retains task-relevant parameters to reduce inference overhead. Inspired by the clustering patterns of hidden states and activations in LLMs, SEAP identifies task-specific expert activation patterns and prunes the model while preserving task performance and enhancing computational efficiency. Experimental results demonstrate that SEAP significantly reduces computational overhead while maintaining competitive accuracy. Notably, at 50% pruning, SEAP surpasses both WandA and FLAP by over 20%, and at 20% pruning, it incurs only a 2.2% performance drop compared to the dense model. These findings highlight SEAP's scalability and effectiveness, making it a promising approach for optimizing large-scale LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。