StreamFlow让模型动态获取历史视觉信息,提升视频理解效率与准确性。
StreamFlow: Dynamic Memory Flows for Streaming Video Understanding

- 引入轻量级动态记忆机制,过滤冗余视频内容再编码
- 在StreamingBench上达67.73%准确率,视觉注意力得分提升59.1%
- 适合需要高效处理长视频流的实时应用开发者
流式视频理解要求多模态大语言模型在严格因果性和有限内存约束下持续保留相关证据。现有方法受限:基于模型的方法需侵入式主干更新,基于记忆的方法对时间冗余内容消耗大量视觉编码计算,并依赖僵化的历史访问。为此,我们提出StreamFlow,一种高效的视觉记忆框架,实现对历史视觉信息的动态按需访问。StreamFlow结合轻量级动态中期记忆(在视觉编码前过滤时间冗余)与潜在长期记忆(将历史视频内容压缩为可后续推理访问的视觉潜变量)。生成阶段,通过注意力引导的检索机制,在模型对视觉证据依赖减弱时注入相关潜变量。StreamFlow在StreamingBench上达到67.73%整体准确率,相较基线提升视觉注意力分数(VAS)59.1%,同时端到端延迟和峰值内存分别降低50.4%和21.1%,实现更精准且高效的视觉推理。
原文摘要 · Abstract (English)
Streaming video understanding requires multimodal large language models (MLLMs) to preserve relevant evidence from continuously evolving streams under strict causality and bounded memory. Yet existing paradigms remain limited: model-based methods require intrusive backbone updates, while memory-based methods expend substantial visual-encoding computation on temporally redundant content and rely on rigid access to visual history. To address these limitations, we introduce StreamFlow, an efficient visual memory framework that enables dynamic, on-demand access to historical visual information. StreamFlow combines a lightweight, dynamics-aware mid-term memory that filters temporal redundancy before visual encoding with a latent long-term memory that consolidates historical video content into visual latents accessible to subsequent reasoning. During generation, an attention-guided retrieval mechanism injects relevant visual latents when the model's reliance on visual evidence weakens. StreamFlow achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench, while also delivering strong performance on offline long-video benchmarks. Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively, enabling more visually grounded and efficient reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。