Vista通过场景感知机制实现流式视频问答的高效长时推理。
Vista: Scene-Aware Optimization for Streaming Video Question Answering under Post-Hoc Queries

- 动态划分视频为视觉与时间一致的场景单元
- 场景压缩后存入GPU,全分辨率帧存于CPU,节省内存
- 查询时仅召回相关场景,兼顾效率与完整性
流式视频问答(Streaming Video QA)对多模态大模型提出独特挑战:视频帧按顺序到达,用户问题可在任意时刻提出。现有依赖固定大小内存或简单压缩的方法常导致上下文丢失或内存溢出,难以应对长时、实时场景。我们提出Vista,一种面向流式视频问答的场景感知优化框架,支持连续视频流的高效可扩展推理。其创新体现在三方面:(1) 场景感知分割,动态将帧聚类为时间与视觉连贯的场景单元;(2) 场景感知压缩,将每个场景压缩为紧凑的标记表示并存储于GPU内存以实现基于索引的高效检索,全分辨率帧则卸载至CPU内存;(3) 场景感知召回,接收查询时选择性召回并重新整合相关场景,兼顾效率与完整性。Vista与模型无关,可无缝集成多种视觉-语言骨干网络,在不牺牲延迟和内存效率的前提下实现长上下文推理。在StreamingBench上的大量实验表明,Vista达到当前最优性能,为真实世界流式视频理解建立强基准。
原文摘要 · Abstract (English)
Streaming video question answering (Streaming Video QA) poses distinct challenges for multimodal large language models (MLLMs), as video frames arrive sequentially and user queries can be issued at arbitrary time points. Existing solutions relying on fixed-size memory or naive compression often suffer from context loss or memory overflow, limiting their effectiveness in long-form, real-time scenarios. We present Vista, a novel framework for scene-aware streaming video QA that enables efficient and scalable reasoning over continuous video streams. The innovation of Vista can be summarized in three aspects: (1) scene-aware segmentation, where Vista dynamically clusters incoming frames into temporally and visually coherent scene units; (2) scene-aware compression, where each scene is compressed into a compact token representation and stored in GPU memory for efficient index-based retrieval, while full-resolution frames are offloaded to CPU memory; and (3) scene-aware recall, where relevant scenes are selectively recalled and reintegrated into the model input upon receiving a query, enabling both efficiency and completeness. Vista is model-agnostic and integrates seamlessly with a variety of vision-language backbones, enabling long-context reasoning without compromising latency or memory efficiency. Extensive experiments on StreamingBench demonstrate that Vista achieves state-of-the-art performance, establishing a strong baseline for real-world streaming video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。