通过混合深度与宽度剪枝，实现大模型高效压缩与加速
Compressing LLMs with MoP: Mixture of Pruners
- 迭代式框架融合深度与宽度剪枝，动态选择最优路径
- 在LLaMA-2/3上实现40%压缩下39%延迟降低，性能超越现有方法
- 适用于多模态模型，文本微调可恢复视觉任务性能
大型语言模型（LLMs）的高计算需求催生了减少参数量并加速推理的方法。模型剪枝成为有效策略，但现有方法通常仅关注单一维度——深度或宽度。本文提出MoP（Mixture of Pruners），一种统一两个维度的迭代框架。每轮迭代中，MoP生成两条分支：分别进行深度剪枝和宽度剪枝，并选择表现更优的候选继续推进。在LLaMA-2和LLaMA-3上，MoP实现了结构化剪枝的新前沿，在多种压缩比例下均优于现有方法，且持续优于仅深度或仅宽度剪枝。此外，MoP将结构化剪枝转化为实际速度提升，在40%压缩率下实现端到端延迟降低39%。进一步将MoP扩展至视觉语言模型LLaVA-1.5，显著提升计算效率，并表明仅用文本恢复微调即可恢复视觉任务性能。
原文摘要 · Abstract (English)
The high computational demands of Large Language Models (LLMs) motivate methods that reduce parameter count and accelerate inference. In response, model pruning emerges as an effective strategy, yet current methods typically focus on a single dimension-depth or width. We introduce MoP (Mixture of Pruners), an iterative framework that unifies these dimensions. At each iteration, MoP generates two branches-pruning in depth versus pruning in width-and selects a candidate to advance the path. On LLaMA-2 and LLaMA-3, MoP advances the frontier of structured pruning, exceeding the accuracy of competing methods across a broad set of compression regimes. It also consistently outperforms depth-only and width-only pruning. Furthermore, MoP translates structural pruning into real speedup, reducing end-to-end latency by 39% at 40% compression. Finally, extending MoP to the vision-language model LLaVA-1.5, we notably improve computational efficiency and demonstrate that text-only recovery fine-tuning can restore performance even on visual tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。