通过学习动态缩放矩阵，提升大模型低秩压缩效果
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
- 用可学习的缩放矩阵替代传统SVD分析推导
- 有效降低权重矩阵的内在秩，熵值下降显著
- 适合需要降低推理计算成本的场景
我们提出SigmaScale，一种基于截断奇异值分解（SVD）的大语言模型（LLM）压缩方法，通过学习辅助缩放矩阵 $S$ 来提升压缩性能。不同于传统的解析推导方式，SigmaScale在激活感知的压缩损失下优化定义行和列缩放变换的两组向量。实验表明，学习到的缩放能降低权重矩阵的有效内在秩，表现为有效秩熵的减少，且该减少与压缩损失强相关。在 Llama 3.1 8B Instruct 与 Qwen3-8B 上的测试显示，SigmaScale 在困惑度和零样本基准上均达到与现有先进SVD压缩方法相当的性能。通过使用激活感知的可学习变换，该方法以更灵活的方式适应各层权重结构，为降低模型推理计算开销提供了有效方案。
原文摘要 · Abstract (English)
We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression. Instead of deriving scaling matrices analytically, SigmaScale optimizes two sets of vectors that define diagonal row and column scaling transformations under an activation-aware compression loss. We show that learned scaling lowers the effective intrinsic rank of weight matrices, as reflected by reductions in effective-rank entropy, and that this reduction is strongly correlated with compression loss. Experiments on Llama 3.1 8B Instruct and Qwen3-8B show that SigmaScale is competitive with closely related state-of-the-art SVD-based compression methods across perplexity and zero-shot benchmarks. By using learned activation-aware transformations, SigmaScale explores a more flexible route to low-rank LLM compression by adapting to the structure of individual model weights. The advantage observed in specific tasks makes our approach a valid option for applications requiring a reduced LLM-inference computing cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。