提出Swift-SVD，实现低秩压缩的理论最优与实际高效兼顾。
Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression
- 基于输出激活的协方差聚合，单次特征分解完成快速压缩。
- 在6个LLM、8个数据集上实现最优压缩精度，压缩速度提升3-70倍。
- 支持动态秩分配，适合追求高效低损模型压缩的研究者。
大语言模型部署受限于静态权重和动态键值缓存带来的内存与带宽压力。基于SVD的压缩能降低这些开销，但现有方法存在重建误差不优或理论最优却效率低的问题。本文提出Swift-SVD，一种激活感知、闭式求解的压缩框架，同时保证理论最优性、实际效率与数值稳定性。该方法在批量输入下增量聚合输出激活的协方差，仅需一次特征值分解即可完成层级低秩近似，实现无训练、快速且最优的压缩。通过有效秩分析各层可压缩性，设计兼顾局部重建损失与端到端层重要性的动态秩分配策略。在6个大语言模型和8个数据集上的大量实验表明，Swift-SVD优于现有最佳基线，达到最优压缩精度的同时，端到端压缩时间提速3-70倍。代码已开源：https://github.com/hiahei/Swift-SVD。
原文摘要 · Abstract (English)
The deployment of Large Language Models is constrained by the memory and bandwidth demands of static weights and dynamic Key-Value cache. SVD-based compression provides a hardware-friendly solution to reduce these costs. However, existing methods suffer from two key limitations: some are suboptimal in reconstruction error, while others are theoretically optimal but practically inefficient. In this paper, we propose Swift-SVD, an activation-aware, closed-form compression framework that simultaneously guarantees theoretical optimum, practical efficiency and numerical stability. Swift-SVD incrementally aggregates covariance of output activations given a batch of inputs and performs a single eigenvalue decomposition after aggregation, enabling training-free, fast, and optimal layer-wise low-rank approximation. We employ effective rank to analyze local layer-wise compressibility and design a dynamic rank allocation strategy that jointly accounts for local reconstruction loss and end-to-end layer importance. Extensive experiments across six LLMs and eight datasets demonstrate that Swift-SVD outperforms state-of-the-art baselines, achieving optimal compression accuracy while delivering 3-70X speedups in end-to-end compression time. Our code is available at https://github.com/hiahei/Swift-SVD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。