提出Pivoting Factorization，让大模型低秩压缩更省内存、更快推理。
Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models
- 通过识别主行并用线性组合表示其他行，实现无损压缩。
- 在50%秩下内存减少24.2%，推理速度提升24.6%，性能接近半结构化剪枝。
- 无需重训练即可修复低秩压缩的性能损失，适合部署优化场景。
大语言模型快速发展，推动了高效模型压缩技术的需求以降低内存与计算开销。低秩剪枝因其对所有密度的GPU兼容性受到关注，但其性能常落后于半结构化剪枝，相同密度下困惑度可能翻倍。本文提出一种无监督的新型无损元低秩表示——Pivoting Factorization（PIFA），能自动学习任意低秩表示的紧凑形式，有效消除冗余信息。PIFA通过识别主行（线性无关行），将非主行表示为线性组合，在秩为维度50%时实现24.2%的额外内存节省和24.6%的推理加速。为缓解低秩剪枝导致的性能下降，我们引入一种免重训练的重建方法，最小化误差累积（M）。MPIFA将M与PIFA整合为端到端框架，显著优于现有低秩剪枝方法，性能媲美半结构化剪枝，且在GPU效率与兼容性上更优。代码已开源：https://github.com/biomedical-cybernetics/pivoting-factorization。
原文摘要 · Abstract (English)
The rapid growth of Large Language Models has driven demand for effective model compression techniques to reduce memory and computation costs. Low-rank pruning has gained attention for its GPU compatibility across all densities. However, low-rank pruning struggles to match the performance of semi-structured pruning, often doubling perplexity at similar densities. In this paper, we propose Pivoting Factorization (PIFA), a novel lossless meta low-rank representation that unsupervisedly learns a compact form of any low-rank representation, effectively eliminating redundant information. PIFA identifies pivot rows (linearly independent rows) and expresses non-pivot rows as linear combinations, achieving 24.2% additional memory savings and 24.6% faster inference over low-rank layers at rank = 50% of dimension. To mitigate the performance degradation caused by low-rank pruning, we introduce a novel, retraining-free reconstruction method that minimizes error accumulation (M). MPIFA, combining M and PIFA into an end-to-end framework, significantly outperforms existing low-rank pruning methods, and achieves performance comparable to semi-structured pruning, while surpassing it in GPU efficiency and compatibility. Our code is available at https://github.com/biomedical-cybernetics/pivoting-factorization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。