针对大模型压缩中误差累积问题,提出感知损失的分层压缩方法
LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

- 根据各层对损失的影响分配压缩秩预算
- 在0.6压缩率下WikiText-2上困惑度降至32.57
- 适合追求高精度压缩的大模型部署场景
大语言模型参数规模快速增长,亟需高效压缩技术。低秩压缩因其硬件无关性和兼容性广受青睐,但现有SVD方法多基于局部重建目标,存在两个关键缺陷:压缩秩分配未考虑层间损失敏感度,且局部近似误差会在残差流中传播累积,导致全局偏差放大。为此,我们提出LACE-SVD框架,首先通过估计候选层压缩比带来的校准负对数似然增量,求解预算约束下的秩分配问题;随后通过闭式更新优化压缩模型,并引入传播感知修正机制,降低残差输出模块的层输出差异,以抑制累积误差。实验表明,在0.6压缩率下,该方法在LLaMA-7B上的WikiText-2困惑度为32.57,显著优于Dobi-SVD的46.18。
原文摘要 · Abstract (English)
The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques. As a hardware-agnostic and highly compatible approach, low-rank compression has been widely adopted to reduce both memory footprint and computational cost. However, existing SVD-based methods are still largely driven by local reconstruction objectives, overlooking two critical limitations: rank budgets are often allocated without explicitly considering layer-wise loss sensitivity, and local approximation errors can propagate and accumulate through the residual stream, leading to amplified global deviations from the original model. To address these issues, we propose LACE-SVD, a Loss-Aware SVD framework with Cumulative Error correction for LLM compression. LACE-SVD first estimates the calibration negative-log-likelihood increase induced by candidate layer-wise compression ratios and solves a budget-constrained allocation problem to assign rank budgets. It then refines the compressed model with closed-form local updates and introduces a propagation-aware correction for residual-stream output modules, reducing layer-output discrepancy as a proxy for cumulative error propagation. Experimental results demonstrate that at a high compression ratio (0.6), the WikiText-2 PPL of our method on LLaMA-7B (32.57) is significantly better than that of Dobi-SVD (46.18).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。