HERMES通过分层缓存实现视频流的高效实时理解。
HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding
- 将键值缓存视为多粒度记忆框架，支持视频信息分层存储。
- 相比现有方法，首字响应时间快10倍，视频令牌减少68%仍保持高精度。
- 无需额外计算即可实时响应，适合资源受限的连续视频交互场景。
多模态大模型在离线视频理解方面取得显著进展，但将其能力扩展到流式视频输入仍面临挑战：现有模型难以同时保证稳定的理解性能、实时响应和低显存开销。为此，我们提出HERMES——一种无需训练的新架构，实现对视频流的实时、准确理解。基于对注意力机制的机理分析，我们将键值缓存（KV cache）构想为一个分层记忆框架，可捕捉视频在不同粒度下的信息。推理时，HERMES复用紧凑的KV缓存，在资源受限条件下实现高效流式理解。值得注意的是，用户查询到达时无需额外计算，确保了对连续视频流交互的实时响应，首字传输时间（TTFT）比之前最先进方法快10倍。即使视频帧采样率降低至均匀采样的32%（减少68%），在所有基准测试中仍达到更优或相当的准确率，流式数据集上最高提升达11.4%。
原文摘要 · Abstract (English)
Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated significant improvement in offline video understanding. However, extending these capabilities to streaming video inputs, remains challenging, as existing models struggle to simultaneously maintain stable understanding performance, real-time responses, and low GPU memory overhead. To address this challenge, we propose HERMES, a novel training-free architecture for real-time and accurate understanding of video streams. Based on a mechanistic attention investigation, we conceptualize KV cache as a hierarchical memory framework that encapsulates video information across multiple granularities. During inference, HERMES reuses a compact KV cache, enabling efficient streaming understanding under resource constraints. Notably, HERMES requires no auxiliary computations upon the arrival of user queries, thereby guaranteeing real-time responses for continuous video stream interactions, which achieves 10$\times$ faster TTFT compared to prior SOTA. Even when reducing video tokens by up to 68% compared with uniform sampling, HERMES achieves superior or comparable accuracy across all benchmarks, with up to 11.4% gains on streaming datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。