通过共现词优化，同时剪枝词表和网络通道，提升大模型效率。
COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens
- 基于共现词激活权重，联合剪枝词表与前馈层通道。
- 在0.5B-70B模型上实现参数、显存和延迟大幅降低。
- 兼容标准架构，适合部署于边缘设备和大规模推理场景。
让大语言模型在内存、延迟和部署成本方面更高效，对边缘部署、交互式应用和规模化可持续推理至关重要。剪枝是一种有前景的技术，但现有方法存在局限：宽度剪枝常破坏标准Transformer结构，需定制推理代码；深度剪枝可能导致精度骤降。此外，许多剪枝方法虽对大模型有效，却难以在小语言模型（SLMs）上保持性能。本文提出COMPACT，联合执行两项操作：(i) 剪除稀有词汇以压缩嵌入层和LM头层，(ii) 使用共现词加权的激活值剪枝前馈网络中间通道，使重要性与剪枝后的词分布对齐。COMPACT兼具深度与宽度剪枝优势，包括：部署友好（保持标准Transformer结构）、可扩展适应（平衡词表与FFN剪枝）、高效的剪枝速度，以及显著的内存节省和吞吐量提升。在Qwen、LLaMA和Gemma系列模型（0.5B-70B）上的实验表明，其下游性能达到当前最优，参数、GPU内存和延迟均有显著减少。
原文摘要 · Abstract (English)
Making large language models (LLMs) more efficient in memory, latency, and serving cost is crucial for edge deployment, interactive applications, and sustainable inference at scale. Pruning is a promising technique, but existing pruning methods are limited: width pruning often breaks the standard transformer layout, requiring custom inference code, while depth pruning can cause abrupt accuracy drops. Also, while many pruning approaches are effective against LLMs, they struggle to maintain performance on small language models (SLMs). In this work, we propose COMPACT, which jointly (i) prunes rare vocabulary to shrink embedding/LM head layers and (ii) prunes FFN intermediate channels using common-token-weighted activations, aligning importance with the post-pruning token distribution. COMPACT inherits strengths of both depth and width pruning, such as: deployment-friendliness (keeps a standard transformer architecture), scale-adaptivity (trade off vocab. vs. FFN pruning), competitive pruning times, and strong memory savings alongside throughput gains. Experiments across Qwen, LLaMA, and Gemma families (0.5B-70B) show state-of-the-art downstream performance, with substantial reductions in parameters, GPU memory, and latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。