用物理启发方法优化大模型压缩，提升效率并消除冗余。
Globally optimized SVD compression of LLMs via Fermi-function-based rank selection and gauge fixing
- 用费米函数将离散截断转为连续优化，自动选择每层最优秩。
- 通过参数规范自由度实现无损压缩，进一步减少低秩因子冗余。
- 适合需要高效部署大模型的研究者与工程师。
大型语言模型对计算资源要求极高。通过奇异值分解（SVD）对模型权重进行低秩分解是压缩大模型的有前景方法，但面临层间秩选择困难和参数冗余等问题。本文提出两项受物理学启发的改进：(1) FermiGrad，一种基于费米函数的梯度下降算法，将离散奇异值截断松弛为连续优化，以确定全局最优的层间秩；(2) PivGa，一种额外的无损压缩方法，利用低秩因子参数化中的内在规范自由度进行压缩。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are very demanding in terms of their computational resources. Low-rank decompositions of LLM weights, e.g. via Singular Value Decomposition (SVD), is a promising approach for LLM compression, but presents several practical hurdles, e.g. selecting appropriate layer-wise ranks and getting rid of its parameter redundancy. In this work, we present two physics-inspired improvements to SVD LLM compression: (1) \textbf{FermiGrad}, a gradient-descent algorithm that determines globally optimal layer-wise ranks by relaxing the discrete singular-value truncation into a continuous optimization using the Fermi function; (2) \textbf{PivGa}, an additional \textit{lossless} compression of the low-rank factors that exploits the intrinsic gauge freedom in their parametrization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。