通过最优剪枝理论优化大模型奇异值压缩，不需训练即可提升性能。
SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression
- 将最优脑外科算法引入奇异值空间，直接优化保留的奇异值。
- 在OPT和LLaMA 2-7B上实现更低困惑度，压缩比更高。
- 兼容现有SVD压缩方法，无需重新训练，适合部署优化场景。
大语言模型在多种任务中表现卓越，但其部署受限于巨大的内存与计算需求。基于奇异值分解（SVD）的低秩压缩是一种有效解决方案，但现有方法仅关注如何分解及保留哪些组件。本文提出SVD-Surgeon，一种无需训练的方法，将最优脑外科（OBS）框架引入奇异值基。将每个奇异值视为参数，计算其闭式更新，以二阶近似补偿因截断被移除的奇异值对模型损失的影响。同一分析也给出了选择剪枝对象的显著性指标。由于直接作用于奇异值分解结构，SVD-Surgeon可叠加于现有SVD压缩器之上。应用于领先的SVD方法SVD-LLM，在OPT系列与LLaMA 2-7B上均实现了更好的困惑度-压缩权衡，且无需任何重训练。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their deployment is constrained by substantial memory and compute requirements. Low-rank compression via singular value decomposition (SVD) is an effective remedy, but existing methods focus on how to factorize and which components to keep. We introduce SVD-Surgeon, a training-free method that brings the Optimal Brain Surgeon (OBS) framework to the singular-value basis. Treating each singular value as a parameter, it computes a closed-form update of the retained singular values that compensates, to second order in the model loss, for those removed by truncation. The same analysis yields a saliency for choosing which values to prune. As it operates directly on the singular-value factorization, SVD-Surgeon can be layered on top of existing SVD compressors. Applied to SVD-LLM, a leading SVD-based method, it improves the perplexity-compression trade-off on the OPT family and LLaMA 2-7B without any retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。