用谱统计量提前预测大模型压缩后的性能下降，省去盲目试错的计算开销。
Predicting LLM Compression Degradation from Spectral Statistics
- 通过稳定秩和信息密度构建压缩退化预测指标γ·ρ̄ₛ
- 该指标在注意力层和MLP层的预测相关性分别达0.890和0.839
- 适合需要高效压缩大模型的研究者和部署工程师
矩阵级低秩压缩是降低大语言模型成本的有前景方法，但执行压缩并评估其在语言任务上的表现可能代价高昂。能否在投入计算前预测压缩带来的性能退化？我们系统分析了Qwen3和Gemma3模型家族在四种代表性低秩压缩方法（普通SVD、两种ASVD变体、SVD-LLM）下的表现。发现稳定秩和每参数比特数的信息密度主导性能退化。压缩比与稳定秩的乘积γ·ρ̄ₛ是准确退化的稳健预测因子，在注意力层和MLP层的留一交叉验证皮尔逊相关性分别达到0.890和0.839。我们从标准SVD截断界和Transformer层误差组合机制出发，提供理论解释。这些发现支持‘先预测再压缩’的工作流：从权重计算γ·ρ̄ₛ，预估退化，仅对理想配置投入计算资源。
原文摘要 · Abstract (English)
Matrix-level low-rank compression is a promising way to reduce the cost of large language models, but running compression and evaluating the resulting models on language tasks can be prohibitively expensive. Can compression-induced degradation be predicted before committing to this compute? We systematically analyze the Qwen3 and Gemma3 model families across four representative low-rank compression methods: vanilla SVD, two ASVD variants, and SVD-LLM. We find that stable rank and information density, measured in bits per parameter, dominate performance degradation. The interaction term $γ\cdot \barρ_s$, defined as compression ratio times stable rank, is a robust predictor of accuracy degradation, achieving leave-one-out cross-validation Pearson correlations of $0.890$ for attention layers and $0.839$ for MLP layers. We provide theoretical intuition for why this predictor succeeds by connecting it to standard SVD truncation bounds and error composition mechanisms in transformer layers. These findings enable a predict-then-compress workflow: compute $γ\cdot \barρ_s$ from weights, estimate degradation, and invest compute only in desirable configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。