无需微调，用低秩矩阵补偿压缩模型性能损失
EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation
- 通过特征空间低秩近似，向压缩模型添加可调补偿矩阵
- 3比特压缩的LLaMA3-8B在多个任务上提升超10%准确率
- 兼容多种硬件，适合对推理效率和精度有灵活需求的场景
后训练压缩技术虽能降低大语言模型的内存、延迟和功耗，但常导致显著的精度下降，且受限于硬件与内核支持的压缩格式，部署灵活性不足。本文提出EoRA——一种无需微调的方法，通过在压缩模型中加入低秩矩阵，实现快速提升特定任务性能，并自由权衡精度与计算开销，突破压缩格式限制。EoRA在恢复压缩模型精度方面显著优于现有无微调方法，在3比特压缩的LLaMA3-8B上，分别在ARC-Challenge、MathQA和GSM8K上取得10.84%、6.74%和11.45%的准确率提升。同时引入优化的CUDA内核，使推理加速最高达1.4倍，并通过量化降低内存开销。EoRA为不同用户需求下的压缩模型性能优化提供了即时解决方案，推动大模型更高效灵活部署。代码已开源。
原文摘要 · Abstract (English)
While post-training compression techniques effectively reduce the memory footprint, latency, and power consumption of Large Language Models (LLMs), they often result in noticeable accuracy degradation and remain limited by hardware and kernel constraints that restrict supported compression formats - ultimately reducing flexibility across a wide range of deployment scenarios. In this work, we propose EoRA - a novel $\textbf{fine-tuning-free}$ method that augments compressed LLMs with low-rank matrices, allowing users to rapidly enhance task-specific performance and freely balance the trade-off between accuracy and computational overhead beyond the constraints of compression formats. EoRA consistently outperforms prior fine-tuning-free low rank methods in recovering the accuracy of compressed LLMs, achieving notable accuracy improvements (e.g., $\mathbf{10.84\%}$ on ARC-Challenge, $\mathbf{6.74\%}$ on MathQA, and $\mathbf{11.45\%}$ on GSM8K for LLaMA3-8B compressed to 3-bit). We also introduce an optimized CUDA kernel, accelerating inference by up to 1.4x and reducing memory overhead through quantizing EoRA. Overall, EoRA offers a prompt solution for improving the accuracy of compressed models under varying user requirements, enabling more efficient and flexible deployment of LLMs. Code is available at https://github.com/NVlabs/EoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。