预训练阶段引导权重低秩化，提升模型压缩后精度恢复效果。
Low-Rank Prehab: Preparing Neural Networks for SVD Compression
- 压缩前先微调，让权重自然趋向低秩结构
- 压缩后准确率下降减少40%以上，微调后性能更优
- 适合大模型压缩场景，尤其适配SVD类方法
低秩近似方法（如奇异值分解SVD及其变体）已成为神经网络压缩的有效工具。在该框架中，分解相当于‘手术’，后续微调则如同‘康复’以恢复精度。受外科术前康复启发，本文提出一种预压缩微调阶段——低秩预康复（Low-Rank Prehab），显式引导权重矩阵形成低秩结构，同时保持任务性能。通过在压缩前对模型进行条件化，Prehab使权重朝参数空间中谱紧凑区域迁移，从而实现更平滑的低秩近似与更好恢复能力。在大型语言模型（LLMs）及基于Transformer的架构（包括视觉变换器ViTs）上实验表明，Prehab显著降低压缩后的即时精度损失，并持续提升微调后性能。在多种压缩比下，本方法优于SVD-LLM等先进SVD方法，凸显了为压缩做准备的重要性，而非仅优化压缩和恢复阶段。代码已开源：https://github.com/niqretnuh/PREHAB-SVD
原文摘要 · Abstract (English)
Low-rank approximation methods such as singular value decomposition (SVD) and its variants (e.g., Fisher-weighted SVD, Activation SVD) have recently emerged as effective tools for neural network compression. In this setting, decomposition acts as a "surgical" intervention, followed by fine-tuning that serves as "rehab" to recover accuracy. Inspired by prehabilitation in surgery, we introduce a pre-compression fine-tuning stage, Low-Rank Prehab, that explicitly encourages low-rank structure in weight matrices while preserving task performance. By conditioning the model before SVD, Prehab steers weights toward spectrally compact regions of the parameter space, enabling smoother low-rank approximation and improved recovery. Experiments on large language models (LLMs) and other Transformer-based architectures, including Vision Transformers (ViTs), show that Prehab substantially reduces the immediate accuracy drop after compression and consistently improves post-finetuning performance. Across a wide range of compression ratios, our method outperforms state-of-the-art SVD-based techniques such as SVD-LLM, highlighting the importance of preparing models for compression rather than only improving the compression and recovery stages. Source code is available at https://github.com/niqretnuh/PREHAB-SVD
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。