提出新方法提升大模型低秩分解精度，效果比现有最优方法高20%-40%。
Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation
- 基于二阶海森矩阵信息，在模型空间中实现双向白化分解
- 相比仅考虑输入信息的方法，分解精度提升20%-40%
- 适用于追求高精度低秩压缩的大模型部署与微调场景
低秩分解已成为大语言模型微调与推理中的关键问题。通过奇异值分解（SVD），权重矩阵可被最优地分解为低秩空间。以往常见做法是在激活白化空间中进行分解，取得良好效果。本文提出最优脑分解大模型（OBD-LLM），在模型空间中利用二阶海森信息研究分解问题。通过严格的克罗内克-因式分解，我们证明分解需同时考虑层的输入与输出信息，相较仅考虑输入的方法显著提升分解质量。所提损失感知分解方法对权重矩阵实施双向白化，实现语言模型权重的闭式最优分解。令人瞩目的是，其性能较现有最优方法SVD-LLM提升约20%-40%。
原文摘要 · Abstract (English)
Low-rank decomposition has emerged as an important problem in Large Language Model (LLM) fine-tuning and inference. Through Singular Value Decomposition (SVD), the weight matrix can be factorized into low-rank spaces optimally. Previously, a common practice was to decompose the weight in the activation-whitened space, and then achieve satisfying results. In this work, we propose Optimal Brain Decomposition LLM (OBD-LLM), which studies the decomposition problem in the model space by utilizing second-order Hessian information. Through a rigorous Kronecker-factorization of the Hessian, we show that the decomposition needs to consider both input and output information of the layer, and achieves much better decomposition results compared to input only method. Our loss-aware decomposition method involves a bi-directional whitening on the weight matrix. As a result, OBD-LLM is a closed-form solution for the optimal decomposition of weights in the language model. Remarkably, we achieve ~20-40\% better results than previous state-of-the-art decomposition methods, the SVD-LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。