用动态剪枝和压缩记忆,让视频模型高效理解长视频。
CacheFlow: Compressive Streaming Memory for Efficient Long-Form Video Understanding
- 每帧实时删减冗余图像块,只保留关键信息
- 处理效率提升87%,仍能精准回答长视频问题
- 无需训练,可直接接入现有视频模型
长时视频问答(VQA)使当前视觉语言模型(VLMs)面临挑战,因注意力和键值(KV)缓存随运行时间增长,导致推理成本高或仅能关注局部窗口。我们提出CacheFlow,一种无需训练的流水线,结合动态令牌剪枝(DTD)与压缩长期记忆。DTD通过与前一帧的余弦相似度在线剪枝每帧图像块,剩余块被封装为固定大小的块。这些块在处理过程中,其键由小型循环编码器总结形成检索索引,而完整键值对则被卸载并后续重新激活以保持答案保真度。推理时,基于共识的检索机制仅召回最相关的前K个块,并同时关注检索到的上下文和本地内容,实现精准长程推理。CacheFlow即插即用、架构无关,无需微调。在离线与流式VQA基准测试中,其性能优于现有强基线,同时处理的令牌量减少高达87%。该双策略使VLM兼具高效性与上下文感知能力,为实际长时视频理解铺平道路。
原文摘要 · Abstract (English)
Long-form video question answering (VQA) overwhelms current vision-language models (VLMs) because attention and key-value (KV) caches grow with runtime, forcing either expensive inference or near-sighted sliding windows. We introduce CacheFlow, a training-free pipeline that pairs Dynamic Token Dropping (DTD) with a compressive long-term memory. DTD prunes per-patch tokens online via cosine similarity to the previous frame, and surviving tokens are packed into fixed-size blocks. This online, per-frame processing makes our approach fundamentally suited for live streaming VQA. As blocks are processed, each one's keys are summarized by a tiny recurrent encoder to form a retrieval index, while the block's full KV pairs are offloaded and later rehydrated for generation, preserving answer fidelity. At inference, a consensus-based retrieval mechanism retrieves only the Top-K most relevant blocks and attends over both the retrieved and local context for precise, long-range reasoning. CacheFlow is drop-in, architecture-agnostic, and requires no fine-tuning. Experiments on both offline and streaming VQA benchmarks demonstrate that CacheFlow outperforms current strong baselines, while processing up to 87% less tokens. Our dual approach enables VLMs to be both efficient and context-aware, paving the way for practical long-form video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。