1-bit量化压缩多模态大模型的键值缓存，大幅提速不丢性能。
CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMs
- 采用1比特量化+新校准方法，针对KV缓存特性优化
- 在InternVL上实现10倍推理吞吐提升
- 即插即用，适配各类多模态大模型
多模态大语言模型（MLLMs）在多种应用中表现优异，但部署时的计算开销仍是关键瓶颈。虽然键值（KV）缓存通过牺牲内存换取计算效率，但庞大的缓存内存占用显著降低吞吐量，限制了在内存受限的GPU设备上的长时间运行。为此，我们提出CalibQuant，一种简单高效的视觉量化策略，可大幅降低内存与计算开销。具体而言，CalibQuant引入极端的1比特量化方案，并结合专为KV缓存内在模式设计的新后缩放与校准技术，在不损失模型性能的前提下保障高效率。借助Triton进行运行时优化，在InternVL模型上实现10倍吞吐量提升。该方法无需修改架构，可无缝集成至多种现有MLLMs。大量实验验证，本方法显著减少内存使用，同时保持计算效率和多模态能力。代码已公开于https://github.com/insuhan/calibquant。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance across diverse applications. However, their computational overhead during deployment remains a critical bottleneck. While Key-Value (KV) caching effectively trades memory for computation to enhance inference efficiency, the growing memory footprint from extensive KV caches significantly reduces throughput and restricts prolonged deployment on memory-constrained GPU devices. To address this challenge, we propose CalibQuant, a simple yet highly effective visual quantization strategy that drastically reduces both memory and computational overhead. Specifically, CalibQuant introduces an extreme 1-bit quantization scheme, complemented by novel post-scaling and calibration techniques tailored to the intrinsic patterns of KV caches, thereby ensuring high efficiency without compromising model performance. Leveraging Triton for runtime optimization, we achieve a 10x throughput increase on InternVL models. Our method is designed to be plug-and-play, seamlessly integrating with various existing MLLMs without requiring architectural changes. Extensive experiments confirm that our approach significantly reduces memory usage while maintaining computational efficiency and preserving multimodal capabilities. Codes are available at https://github.com/insuhan/calibquant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。