提出新方法压缩大模型，更准确评估参数重要性
Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models
- 用改进的奇异值分解方法，同时考虑参数相关性与独立性
- 在MMLU上20倍压缩率下性能领先其他方法3%至6%
- 适合追求高精度模型压缩的研究者和工程应用
Fisher信息是刻画神经网络参数敏感性的基础概念。但对大模型而言，完整使用观测Fisher信息计算成本过高，因此多数方法采用简单的对角近似。虽高效，却忽略参数间相关性，常导致下游任务性能下降。本文提出广义Fisher加权奇异值分解（GFWSVD），一种后训练的大语言模型压缩技术，可同时考虑Fisher信息矩阵的对角与非对角元素，更准确反映参数重要性。为提升可扩展性，我们引入一种针对观测Fisher信息的可扩展克罗内克因子化近似算法。实验表明该方法在大模型压缩中有效：例如在MMLU基准上，20倍压缩率下，相比基于对角近似的FWSVD提升5个百分点，比SVD-LLM提升3个百分点，比ASVD提升6个百分点。
原文摘要 · Abstract (English)
The Fisher information is a fundamental concept for characterizing the sensitivity of parameters in neural networks. However, leveraging the full observed Fisher information is too expensive for large models, so most methods rely on simple diagonal approximations. While efficient, this approach ignores parameter correlations, often resulting in reduced performance on downstream tasks. In this work, we mitigate these limitations and propose Generalized Fisher-Weighted SVD (GFWSVD), a post-training LLM compression technique that accounts for both diagonal and off-diagonal elements of the Fisher information matrix, providing a more accurate reflection of parameter importance. To make the method tractable, we introduce a scalable adaptation of the Kronecker-factored approximation algorithm for the observed Fisher information. We demonstrate the effectiveness of our method on LLM compression, showing improvements over existing compression baselines. For example, at a 20 compression rate on the MMLU benchmark, our method outperforms FWSVD, which is based on a diagonal approximation of the Fisher information, by 5 percent, SVD-LLM by 3 percent, and ASVD by 6 percent compression rate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。