用单个语义载体压缩每帧,实现视频实时理解的高效推理。
VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carriers
- 每帧仅用一个语义载体替代大量视觉标记,降低计算开销。
- 相比原版模型提速近5倍,显存稳定在约18GB不随视频变长增加。
- 适合需要低延迟、长视频实时处理的应用场景。
本文提出VideoScan,一种面向实时视频交互的高效视觉语言模型(VLM)推理框架,能有效理解并保持流式视频输入,同时实现快速准确响应。视频理解长期面临计算开销大的挑战,尤其在长时或实时应用中,源于视觉标记数量庞大。VideoScan通过每帧使用单一语义载体标记,在双阶段推理流程(预填充与解码)中逐步降低计算与内存负担。语义载体嵌入由帧级视觉特征优化聚合得到,确保表示紧凑且语义丰富;对应的键值对经训练保留前序帧上下文语义,实现高效内存管理而不损失时间连贯性。推理过程中,每帧视觉标记仅在预填充阶段处理一次,解码阶段即丢弃,消除重复计算。实验证明,基于该方法的LLaVA-Video在离线与在线基准上,相比原版提速约5倍,较此前高效流式视频理解方法提速1.29倍,性能保持竞争力,且显存稳定在约18GB,不受视频长度影响。
原文摘要 · Abstract (English)
This paper introduces VideoScan, an efficient vision-language model (VLM) inference framework designed for real-time video interaction that effectively comprehends and retains streamed video inputs while delivering rapid and accurate responses. A longstanding challenge in video understanding--particularly for long-term or real-time applications--stems from the substantial computational overhead caused by the extensive length of visual tokens. To address this, VideoScan employs a single semantic carrier token to represent each frame, progressively reducing computational and memory overhead during its two-phase inference process: prefilling and decoding. The embedding of the semantic carrier token is derived from an optimized aggregation of frame-level visual features, ensuring compact yet semantically rich representations. Critically, the corresponding key-value pairs are trained to retain contextual semantics from prior frames, enabling efficient memory management without sacrificing temporal coherence. During inference, the visual tokens of each frame are processed only once during the prefilling phase and subsequently discarded in the decoding stage, eliminating redundant computations. This design ensures efficient VLM inference even under stringent real-time constraints. Comprehensive experiments on diverse offline and online benchmarks demonstrate that LLaVA-Video, supported by our method, achieves up to $\sim 5\times$ and $1.29\times$ speedups compared to its original version and previous efficient streaming video understanding approaches, respectively. Crucially, these improvements are attained while maintaining competitive performance and ensuring stable GPU memory consumption (consistently $\sim 18$GB, independent of video duration).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。