提出新方法压缩多模态模型缓存,速度提升1.69倍且内存降80%。
Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach
- 从频域能量分布出发,用低通滤波提取主成分特征。
- 识别并保留关键异常值键值对,避免性能下降。
- 适配FlashAttention,适合高效率多模态推理场景。
多模态大语言模型因视觉输入长度增长导致键值缓存(KV Cache)规模过大,推理开销显著。现有压缩方法主要依赖注意力分数,不兼容高效注意力核(如FlashAttention),且忽略值向量对注意力输出的贡献。本文从键值矩阵分布角度重新审视压缩问题:首先发现多模态键值矩阵的频域能量主要集中于低频,通过低通滤波提取主成分;进一步发现移除与主成分偏差大的键值对(即异常值键值对,Outlier KVs)会导致明显性能下降。由于异常值键值对更可能包含关键特征,本文提出FlashCache框架,包含频域引导的异常值识别模块,优先保留显著偏离主成分的键值对,并设计动态预算分配模块,自适应调整各层缓存大小以保留更多异常值键值对。在多个多模态大模型和基准测试上验证,FlashCache优于当前最优方法,实现最高1.69倍加速、KV内存降低80%的同时保持任务性能不变。
原文摘要 · Abstract (English)
Multimodal large language models suffer from substantial inference overhead since multimodal KV Cache grows proportionally with the visual input length. Existing multimodal KV Cache compression methods mostly rely on attention score to reduce cache size, which makes them are incompatible with established efficient attention kernels (e.g., FlashAttention) and ignores the contribution of value vectors to the attention output. In this work, we revisit multimodal KV Cache compression from the perspective of the KV matrices' distribution. First, we observe that frequency-domain energy of multimodal KV matrices is predominantly concentrated in low-frequency and extract this principal energy via a low-pass filter. Further, we find that removing KV pairs that deviate substantially from this principal energy leads to a pronounced performance drop, which we define as Outlier KVs. Considering Outlier KVs are more likely to encode features critical for inference, we propose FlashCache, a frequency-domain-guided, Outlier-KV-aware KV Cache compression framework. First, we introduce an Outlier KV Recognition Module that models the principal component of multimodal KV matrices in the frequency domain and preferentially retains KV pairs that significantly deviate from it. Furthermore, Dynamic Budget Allocation Module is designed to adaptively determine the per-layer KV Cache size to retain more Outlier KVs. Experiments on multiple MLLMs and benchmarks demonstrate that FlashCache outperforms state-of-the-art multimoal KV compression methods, achieving up to 1.69 times faster decoding with 80% lower KV memory usage while maintaining task performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。