让长视频实时理解更高效，用双记忆模块减少计算负担。
Flash-VStream: Efficient Real-Time Understanding for Long Video Streams
- 设计双记忆模块：低容量存全局时序信息，高容量按需提取细节。
- 在多个长视频数据集上实现领先性能，推理延迟显著降低。
- 适合需要实时处理超长视频的场景，如监控、自动驾驶。
得益于大语言模型和跨模态对齐的发展，现有多模态大模型在图像和短视频理解上表现优异。然而，长视频因上下文过长，导致计算与内存开销巨大，现有方法仍将其等同于短视频处理，效率低下且难以扩展至更长视频。为此，我们提出 Flash-VStream，一种可高效处理极长视频并实时响应用户查询的视频语言模型。核心是 Flash Memory 模块：低容量上下文记忆用于聚合长时序信息并建模信息密度分布；高容量增强记忆则根据该分布检索细节空间信息。相比现有模型，Flash-VStream 显著降低推理延迟。在 EgoSchema、MLVU、LVBench、MVBench 与 Video-MME 等长视频及综合基准上，实验验证了其卓越性能与高效性。代码已开源。
原文摘要 · Abstract (English)
Benefiting from the advances in large language models and cross-modal alignment, existing multimodal large language models have achieved prominent performance in image and short video understanding. However, the understanding of long videos is still challenging, as their long-context nature results in significant computational and memory overhead. Most existing work treats long videos in the same way as short videos, which is inefficient for real-world applications and hard to generalize to even longer videos. To address these issues, we propose Flash-VStream, an efficient video language model capable of processing extremely long videos and responding to user queries in real time. Particularly, we design a Flash Memory module, containing a low-capacity context memory to aggregate long-context temporal information and model the distribution of information density, and a high-capacity augmentation memory to retrieve detailed spatial information based on this distribution. Compared to existing models, Flash-VStream achieves significant reductions in inference latency. Extensive experiments on long video benchmarks and comprehensive video benchmarks, i.e., EgoSchema, MLVU, LVBench, MVBench and Video-MME, demonstrate the state-of-the-art performance and outstanding efficiency of our method. Code is available at https://github.com/IVGSZ/Flash-VStream.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。