让量化和低秩矩阵各司其职，提升大模型压缩效果
Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
- 低秩组件专抓敏感权重，避免干扰量化
- 在4位以下设置下，困惑度降低12.3%，准确率提升5.6%
- 适合追求高效推理的模型压缩研究者
将权重重构成量化与低秩成分（$\mathbf{W} \approx \mathbf{Q} + \mathbf{L}\mathbf{R}$）是压缩大语言模型（LLMs）的常用方法。现有联合优化方法通过迭代交替进行量化与低秩近似，但常偏向一方，导致次优分解，未能充分发挥各自优势。本文提出异常值驱动的低秩初始化（ODLRI），使低秩组件专门捕获对激活敏感的权重。该结构化分解有效缓解异常值对量化的负面影响，实现量化与低秩近似的更优平衡。在 Llama2（7B, 13B, 70B）、Llama3-8B 与 Mistral-7B 上的实验表明，将 ODLRI 纳入联合优化框架后，激活感知误差持续下降，量化尺度最小化，且在低比特设置下困惑度降低、零样本准确率提升。
原文摘要 · Abstract (English)
Decomposing weight matrices into quantization and low-rank components ($\mathbf{W} \approx \mathbf{Q} + \mathbf{L}\mathbf{R}$) is a widely used technique for compressing large language models (LLMs). Existing joint optimization methods iteratively alternate between quantization and low-rank approximation. However, these methods tend to prioritize one component at the expense of the other, resulting in suboptimal decompositions that fail to leverage each component's unique strengths. In this work, we introduce Outlier-Driven Low-Rank Initialization (ODLRI), which assigns low-rank components the specific role of capturing activation-sensitive weights. This structured decomposition mitigates outliers' negative impact on quantization, enabling more effective balance between quantization and low-rank approximation. Experiments on Llama2 (7B, 13B, 70B), Llama3-8B, and Mistral-7B demonstrate that incorporating ODLRI into the joint optimization framework consistently reduces activation-aware error, minimizes quantization scale, and improves perplexity and zero-shot accuracy in low-bit settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。