arXiv:2508.16680cs.LGcs.AI2025-08

通过可学习的修正模块,让压缩后的大模型更接近原模型性能。

CALR: Corrective Adaptive Low-Rank Decomposition for Efficient Large Language Model Layer Compression

  • 用SVD压缩层,并加一个可训练的纠错模块恢复损失功能。
  • 压缩率26.93%~51.77%,性能保留59.45%~90.42%。
  • 适合资源受限环境部署,尤其关注性能保持的场景。

大型语言模型因规模庞大和计算需求高,部署面临挑战。模型压缩技术对使其适用于资源受限环境至关重要。主流方法是通过奇异值分解(SVD)进行低秩分解以减少参数量,但标准SVD仅最小化矩阵重建误差,常导致模型功能性能显著下降。这是因为现有方法未能有效纠正压缩过程中的功能信息损失。为此,我们提出纠正性自适应低秩分解(CALR),一种双路径压缩方法:主路径采用SVD压缩层,辅以并行、可学习的低秩修正模块,专门训练以恢复功能残差误差。在SmolLM2-135M、Qwen3-0.6B和Llama-3.2-1B上的实验表明,CALR可实现26.93%至51.77%的参数减少,同时保留59.45%至90.42%的原始模型性能,持续优于LaCo、ShortGPT和LoSparse。CALR的成功表明,将功能信息损失视为可学习信号是一种高效压缩范式,有助于构建更小、更高效的LLM,推动其在实际应用中的可及性与部署。

原文摘要 · Abstract (English)

Large Language Models (LLMs) present significant deployment challenges due to their immense size and computational requirements. Model compression techniques are essential for making these models practical for resource-constrained environments. A prominent compression strategy is low-rank factorization via Singular Value Decomposition (SVD) to reduce model parameters by approximating weight matrices. However, standard SVD focuses on minimizing matrix reconstruction error, often leading to a substantial loss of the model's functional performance. This performance degradation occurs because existing methods do not adequately correct for the functional information lost during compression. To address this gap, we introduce Corrective Adaptive Low-Rank Decomposition (CALR), a two-component compression approach. CALR combines a primary path of SVD-compressed layers with a parallel, learnable, low-rank corrective module that is explicitly trained to recover the functional residual error. Our experimental evaluation on SmolLM2-135M, Qwen3-0.6B, and Llama-3.2-1B, demonstrates that CALR can reduce parameter counts by 26.93% to 51.77% while retaining 59.45% to 90.42% of the original model's performance, consistently outperforming LaCo, ShortGPT, and LoSparse. CALR's success shows that treating functional information loss as a learnable signal is a highly effective compression paradigm. This approach enables the creation of significantly smaller, more efficient LLMs, advancing their accessibility and practical deployment in real-world applications.

模型压缩低秩分解大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。