一种统一框架，让大模型压缩保持形状且性能更优。
NoWag: A Unified Framework for Shape Preserving Compression of Large Language Models
- 基于权重与激活归一化，统一设计压缩算法
- 70B模型压缩后仍保持高精度，优于现有方法
- 适合部署大模型的资源受限场景
大语言模型在自然语言处理任务中表现卓越，但计算和内存开销巨大，限制了其在资源受限环境中的部署。为解决此问题，我们提出 NoWag（归一化权重与激活引导压缩），一种用于单次执行、形状保持型压缩的统一框架。将 NoWag 应用于 Llama-2（7B、13B、70B）和 Llama-3（8B、70B）模型，分别采用向量量化（NoWag-VQ）和非结构化/半结构化剪枝（NoWag-P）两种主流形状保持技术。结果表明，NoWag-VQ 显著优于当前最先进的单次向量量化方法，而 NoWag-P 在性能上可与领先剪枝技术媲美。这些发现揭示了不同压缩范式间的内在共性，为未来研究指明方向。代码已开源：https://github.com/LawrenceRLiu/NoWag。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit remarkable performance across various natural language processing tasks but suffer from immense computational and memory demands, limiting their deployment in resource-constrained environments. To address this challenge, we propose NoWag (Normalized Weight and Activation Guided Compression), a unified framework for one-shot shape preserving compression algorithms. We apply NoWag to compress Llama-2 (7B, 13B, 70B) and Llama-3 (8B, 70B) models using two popular shape-preserving techniques: vector quantization (NoWag-VQ) and unstructured/semi-structured pruning (NoWag-P). Our results show that NoWag-VQ significantly outperforms state-of-the-art one-shot vector quantization methods, while NoWag-P performs competitively against leading pruning techniques. These findings highlight underlying commonalities between these compression paradigms and suggest promising directions for future research. Our code is available at https://github.com/LawrenceRLiu/NoWag
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。