用SVD压缩视觉语言模型的键值权重，显著降耗提效。
QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models
- 对QKV权重矩阵联合做SVD分解，动态分配秩以平衡精度与效率。
- 在低精度下实现超10%精度提升，同时降低内存和计算开销。
- 适合资源受限设备实时部署，开源代码可复现。
视觉语言模型（VLMs）在图像描述和视觉问答等任务中至关重要，但其高计算成本源于庞大的内存占用和处理时间，限制了可扩展性和实时应用。本文提出对查询（Q）、键（K）和值（V）权重矩阵联合使用奇异值分解（SVD），以减少键值缓存大小和计算开销。我们进一步设计了一种高效的秩分配策略，根据对模型精度的影响动态调整SVD秩，显著降低内存使用和计算成本。最后，将量化技术应用于VLM权重和激活值，构建出高效模型。相比仅使用量化或SVD的先前方法，本方案在硬件成本更低的情况下实现了超过10%的精度提升，更适用于资源受限设备的实时部署。代码已开源：https://github.com/SAI-Lab-NYU/QSVD。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) are integral to tasks such as image captioning and visual question answering, but their high computational cost, driven by large memory footprints and processing time, limits their scalability and real-time applicability. In this work, we propose leveraging Singular-Value Decomposition (SVD) over the joint query (Q), key (K), and value (V) weight matrices to reduce KV cache size and computational overhead. We in addition introduce an efficient rank allocation strategy that dynamically adjusts the SVD rank based on its impact on VLM accuracy, achieving a significant reduction in both memory usage and computational cost. Finally, we extend this approach by applying quantization to both VLM weights and activations, resulting in a highly efficient VLM. Our method outperforms previous approaches that rely solely on quantization or SVD by achieving more than $10\%$ accuracy improvement while consuming less hardware cost, making it better for real-time deployment on resource-constrained devices. We open source our code at \href{https://github.com/SAI-Lab-NYU/QSVD}{\texttt{https://github.com/SAI-Lab-NYU/QSVD}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。