揭示大模型压缩中张量分解的局限性，明确其适用边界。
Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression
- 系统评估张量分解在密集与MoE架构中的压缩效果
- 发现张量分解假设的共享子空间与模型实际表示不匹配
- 为大规模部署提供可信赖的压缩方法选择依据
后训练压缩对在资源受限环境下部署大语言模型至关重要。张量分解因其能有效压缩Transformer权重结构而成为有前景的方向。然而，现有研究多在狭窄场景下评估，未能明确该方法在大规模部署中的有效性。本文系统评估了张量压缩在密集与MoE架构中的表现，结合实证与理论分析，揭示了张量分解所依赖的共享子空间假设与现代大模型学习到的异构表示之间存在根本性不匹配，从而明确了其实际限制，并厘清了其在大规模部署中的可行角色。代码已开源：https://github.com/brain-lab-research/TT-LLM。
原文摘要 · Abstract (English)
Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints. Tensor decompositions have emerged as a promising direction, offering compact parameterizations well suited to Transformer weight structures. However, existing studies evaluate these methods in narrow settings, leaving unclear whether tensorization is effective at large-scale deployment. We systematically evaluate tensor compression across dense and MoE architectures, establishing performance trade-offs grounded in both empirical analysis and theoretical analysis. We identify a fundamental mismatch between the shared subspaces assumed by tensor decompositions and the heterogeneous representations learned by modern LLMs, thereby delineating their practical limits and clarifying their viable role in large-scale deployment. The code is available at https://github.com/brain-lab-research/TT-LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。