针对大模型压缩中低秩分解误差不均的问题，提出双层级优化方法。
Beyond Uniform SVD:Dual-Level Optimization across Columns and Modules for LLM Compression
- 按列保留高误差部分，仅对低误差列做低秩近似
- 模块级自适应分配压缩比例，基于扰动偏差优化
- 无需训练，比现有方法更高效，适合模型压缩场景
低秩分解，尤其是奇异值分解（SVD），是缓解大语言模型（LLMs）存储与计算开销的关键技术。然而，现有SVD方法忽视了参数矩阵中不同组件的分解误差存在显著差异，常导致近似效果不佳。此外，缺乏对单个权重矩阵重要性的直接评估指标。为此，我们提出Duo-SVD（双层级优化SVD），一种无需训练的新型框架，在列级和模块级同步优化。首先，引入列保持策略，显式保留高分解误差的列，仅对低误差列应用低秩近似。其次，在模块级采用模块自适应分配策略，将压缩比例分配建模为基于扰动引起的模型偏差的全局约束优化问题。大量实验表明，Duo-SVD在各项指标上持续优于最先进的SVD基基线及结构化剪枝方法，确立了其在高效大模型压缩中的优越地位。
原文摘要 · Abstract (English)
Low-rank decomposition, particularly Singular Value Decomposition (SVD), is a pivotal technique for mitigating the storage and computational demands of Large Language Models (LLMs). However, prevalent SVD-based approaches overlook the critical phenomenon that decomposition errors exhibit significant disparity across different components of the parameter matrix, often leading to suboptimal approximation. Furthermore, existing methods lack a direct metric to evaluate the importance of individual weight matrices. To address these limitations, we propose Duo-SVD (Dual-level Optimization SVD), a novel training-free framework that synergizes optimization at both the column and the module levels. First, Duo-SVD incorporates a Column-Preserving Strategy that explicitly retains columns exhibiting high decomposition errors, while applying low-rank approximation solely to those with lower errors. Second, at the module level, we employ a Module-Adaptive Allocation Strategy that formulates ratio allocation as a global constrained optimization problem based on perturbation-induced model deviation. Extensive experiments demonstrate that Duo-SVD consistently outperforms state-of-the-art SVD-based baselines and structured pruning methods, establishing it as a superior paradigm for efficient LLM compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。