通过双信号预判优化视频注意力缓存，大幅降低内存占用且不损失准确率。
Dual-Signal Adaptive KV-Cache Optimization for Long-Form Video Understanding in Vision-Language Models
- 用光流与显著性检测提前判断帧间冗余和关键区域，主动管理内存。
- 在LLaVA 1.6上实现2.20倍内存压缩，保持100%的生成准确率。
- 适合需要高效处理长视频的开发者，尤其在消费级硬件上表现优异。
视觉语言模型（VLMs）在处理长视频时面临严重的内存瓶颈，因键值（KV）缓存随序列长度线性增长。现有方法多采用反应式淘汰策略，在计算完整注意力矩阵后才丢弃标记，造成大量计算浪费。本文提出Sali-Cache，一种基于先验优化的自适应缓存框架，通过主动内存管理实现双信号感知：基于光流分析的时序滤波器用于检测帧间冗余，结合显著性检测的空间滤波器识别视觉重要区域。该方法在进入昂贵的注意力运算前即完成内存分配决策。在LLaVA 1.6架构上的实验表明，本方法实现了2.20倍的有效内存压缩比，同时在BLEU、ROUGE-L和Exact Match指标上保持100%准确率。在相同内存预算下，Sali-Cache能有效保留更长时间跨度的上下文特征，且不降低模型性能，使长视频内容在消费级硬件上得以高效处理。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) face a critical memory bottleneck when processing long-form video content due to the linear growth of the Key-Value (KV) cache with sequence length. Existing solutions predominantly employ reactive eviction strategies that compute full attention matrices before discarding tokens, resulting in substantial computational waste. We propose Sali-Cache, a novel a priori optimization framework that implements dual-signal adaptive caching through proactive memory management. By integrating a temporal filter based on optical flow analysis for detecting inter-frame redundancy and a spatial filter leveraging saliency detection for identifying visually significant regions, Sali-Cache intelligently manages memory allocation before entering computationally expensive attention operations. Experimental evaluation on the LLaVA 1.6 architecture demonstrates that our method achieves a 2.20x compression ratio in effective memory usage while maintaining 100% accuracy across BLEU, ROUGE-L, and Exact Match metrics. Furthermore, under identical memory budget constraints, Sali-Cache preserves context-rich features over extended temporal durations without degrading model performance, enabling efficient processing of long-form video content on consumer-grade hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。