arXiv:2602.03051cs.CL2026-02被引 8

解决大模型压缩中误差累积问题,提升压缩后性能

SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression

  • 联合优化层内重建与层间误差补偿,减少误差传播
  • 在多个模型上实现无微调下性能超越现有方法
  • 适合需要高效压缩且保持高精度的场景

大语言模型参数规模快速增长,对高效压缩技术需求迫切。低秩压缩因硬件无关性和高度兼容性被广泛应用,但现有方法通常独立压缩每层以最小化层内重建误差,忽略了误差在网络中传播和累积的问题,导致整体偏差放大。为此,我们提出自适应误差抑制奇异值分解(SAES-SVD)框架,联合优化层内重建与层间误差补偿。该框架包含两个新组件:(1) 累积误差感知层压缩(CEALC),将压缩目标设为局部重建与加权累积误差补偿之和,基于二阶激活统计推导出闭式低秩解,显式对齐各层输出与全精度对应结果以补偿累积误差;(2) 自适应协同误差抑制(ACES),自动调节权重系数以增强压缩目标的低秩结构。具体地,通过最大化压缩层输出的Frobenius范数与压缩目标范数之比,在固定秩下确保秩预算有效利用。在多个大模型架构和任务上的大量实验表明,无需微调或混合秩策略,SAES-SVD始终显著提升压缩后性能。

原文摘要 · Abstract (English)

The rapid growth in the parameter scale of large language models (LLMs) has created a high demand for efficient compression techniques. As a hardware-agnostic and highly compatible technique, low-rank compression has been widely adopted. However, existing methods typically compress each layer independently by minimizing per-layer reconstruction error, overlooking a critical limitation: the reconstruction error propagates and accumulates through the network, which leads to amplified global deviations from the full-precision baseline. To address this, we propose Self-Adaptive Error Suppression SVD (SAES-SVD), a LLMs compression framework that jointly optimizes intra-layer reconstruction and inter-layer error compensation. SAES-SVD is composed of two novel components: (1) Cumulative Error-Aware Layer Compression (CEALC), which formulates the compression objective as a combination of local reconstruction and weighted cumulative error compensation. Based on it, we derive a closed-form low-rank solution relied on second-order activation statistics, which explicitly aligns each layer's output with its full-precision counterpart to compensate for accumulated errors. (2) Adaptive Collaborative Error Suppression (ACES), which automatically adjusts the weighting coefficient to enhance the low-rank structure of the compression objective in CEALC. Specifically, the coefficient is optimized to maximize the ratio between the Frobenius norm of the compressed layer's output and that of the compression objective under a fixed rank, thus ensuring that the rank budget is utilized effectively. Extensive experiments across multiple LLM architectures and tasks show that, without fine-tuning or mixed-rank strategies, SAES-SVD consistently improves post-compression performance.

大模型压缩低秩分解误差抑制SVD

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。