跨层压缩新框架失效，因模型前向传播实际解耦层间关系。
Cross-Layer Subspace Coupling for LLM Compression: A Unifying Framework and Its Empirical Limits
- 统一多层奇异值分解压缩为单一优化问题
- 理论误差降低46%，但下游任务性能显著下降
- 未来应聚焦单层激活重构而非权重重建
基于奇异值分解的大型语言模型压缩方法（如 SVD LLM 与 Basis Sharing）可被统一为一个优化问题。数学证明与 Pythia 模型测试显示，该统一方法在权重重建误差上最高提升46%。然而在实际任务中，困惑度与准确率均大幅下降，劣于标准的逐层压缩方法。作者从机制上解释：尽管捆绑法在数学上耦合相邻层，但变压器的残差流在前向传播中实际上解耦了层间关系。因此，逐层最优比跨层联合优化更重要。论文结论：权重空间重构是跨层压缩的错误目标，未来方法应转向逐层激活重构。
原文摘要 · Abstract (English)
Recent SVD based compression methods for large language models like SVD LLM and Basis Sharing can be unified under one optimization problem. While mathematical proofs and tests on Pythia models show this unified approach improves weight reconstruction error by up to 46% percent it fails in practical tasks. Downstream metrics like perplexity and accuracy severely degrade compared to standard per layer SVD LLM. The authors explain this failure mechanistically. Although the bundle method mathematically couples adjacent layers the transformer residual stream actually decouples them during forward passes. Thus per layer optimality matters more than joint cross layer optimization. The paper concludes that weight space reconstruction is a flawed objective for cross layer compression and future methods must focus on per layer activation reconstruction instead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。