用几何视角统一矩阵与张量分解，揭示语言模型压缩的深层原理。
Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models
- 以子空间为核心，将矩阵/张量分解转化为几何变换。
- 首次系统梳理多类压缩方法的内在关联，打破碎片化研究格局。
- 适合对模型压缩机制、几何建模感兴趣的算法研究者。
面向自然语言处理模型的矩阵与张量引导参数化，在提升系统效率方面具有根本性价值。然而，这两类代数结构与语言模型参数化之间的内在联系仍不清晰，且现有研究数学抽象过重，远离机器学习与自然语言处理的研究语境。这一问题导致近期相关进展更像矩阵/张量与NLP研究中零散组件的堆叠，而非有机统一的方法体系，制约了算法设计。为此，本文提出一种统一分类框架，弥合矩阵/张量压缩方法与机器学习及自然语言处理中的模型压缩概念之间的鸿沟。我们采用线性代数中的基础概念——子空间（subspace），该概念也是几何代数的核心，将其作为统摄矩阵/张量与机器学习/自然语言处理概念（如注意力机制）的统一框架。在此基础上，典型矩阵与张量分解算法可被解释为几何变换。最后，我们重新审视近期关于矩阵或张量引导的语言模型压缩文献，重新表述并比较其核心思想，指出现有研究空白与潜在解决方案。
原文摘要 · Abstract (English)
Matrix and tensor-guided parametrization for Natural Language Processing (NLP) models is fundamentally useful for the improvement of the model's systematic efficiency. However, the internal links between these two algebra structures and language model parametrization are poorly understood. Also, the existing matrix and tensor research is math-heavy and far away from machine learning (ML) and NLP research concepts. These two issues result in the recent progress on matrices and tensors for model parametrization being more like a loose collection of separate components from matrix/tensor and NLP studies, rather than a well-structured unified approach, further hindering algorithm design. To this end, we propose a unified taxonomy, which bridges the matrix/tensor compression approaches and model compression concepts in ML and NLP research. Namely, we adopt an elementary concept in linear algebra, that of a subspace, which is also the core concept in geometric algebra, to reformulate the matrix/tensor and ML/NLP concepts (e.g. attention mechanism) under one umbrella. In this way, based on our subspace formalization, typical matrix and tensor decomposition algorithms can be interpreted as geometric transformations. Finally, we revisit recent literature on matrix- or tensor-guided language model compression, rephrase and compare their core ideas, and then point out the current research gap and potential solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。