针对视觉语言模型长序列内存瓶颈，提出自适应2比特量化方法
AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models
- 根据注意力显著性动态分配量化位数，提升压缩效率
- 2比特量化下准确率保持甚至提升，峰值内存降低2.13倍
- 适合大批次、高吞吐的多模态推理场景
视觉语言模型在多模态任务中表现优异，但过长的多模态输入导致键值（KV）缓存过大，引发显著内存占用和I/O瓶颈。现有针对大语言模型的KV量化方法忽略多模态标记的注意力显著性差异，导致性能不佳。本文研究视觉语言模型中的注意力感知标记显著性模式，提出AKVQ-VL。该方法利用提出的文本显著注意力（TSA）和枢轴标记显著注意力（PSA）模式，自适应分配比特预算。此外，实现极低比特量化需有效处理KV张量中的异常值。AKVQ-VL采用沃尔什-哈达玛变换（WHT）构建无异常值的KV缓存，从而降低量化难度。在12个长上下文多模态任务上进行2比特量化评估显示，AKVQ-VL在保持或提升准确率的同时，优于面向大语言模型的方法。该方法可将峰值内存使用减少2.13倍，支持高达3.25倍更大的批处理规模和2.46倍的吞吐量。
原文摘要 · Abstract (English)
Vision-language models (VLMs) show remarkable performance in multimodal tasks. However, excessively long multimodal inputs lead to oversized Key-Value (KV) caches, resulting in significant memory consumption and I/O bottlenecks. Previous KV quantization methods for Large Language Models (LLMs) may alleviate these issues but overlook the attention saliency differences of multimodal tokens, resulting in suboptimal performance. In this paper, we investigate the attention-aware token saliency patterns in VLM and propose AKVQ-VL. AKVQ-VL leverages the proposed Text-Salient Attention (TSA) and Pivot-Token-Salient Attention (PSA) patterns to adaptively allocate bit budgets. Moreover, achieving extremely low-bit quantization requires effectively addressing outliers in KV tensors. AKVQ-VL utilizes the Walsh-Hadamard transform (WHT) to construct outlier-free KV caches, thereby reducing quantization difficulty. Evaluations of 2-bit quantization on 12 long-context and multimodal tasks demonstrate that AKVQ-VL maintains or even improves accuracy, outperforming LLM-oriented methods. AKVQ-VL can reduce peak memory usage by 2.13x, support up to 3.25x larger batch sizes and 2.46x throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。