让视频大模型流式处理突破100帧/秒,实时响应快到50毫秒。
ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inference

- 分阶段协同推理,控制每帧计算与内存开销。
- 单卡A100实现134帧/秒吞吐,首次达到真实流媒体速度。
- 适合需要低延迟视频理解的实时应用开发人员。
流式视频大模型需持续处理视频流并保持低查询延迟,视频摄入吞吐量与查询响应速度对实时部署至关重要。现有方法多聚焦于加速单一模块(如视觉编码、令牌剪枝或键值缓存压缩),却缺乏对系统能否维持实时流性能的深入分析。本文将流式视频大模型推理建模为涵盖视觉预处理、视觉编码、令牌丢弃与大模型预填充/解码的协同流水线。基于此,提出ViCoStream(视频协同流式处理)框架,通过分块执行、CUDA流重叠、视觉令牌控制、有界视觉注意力及查询端检索,有效控制每块计算与内存成本。进一步系统研究瓶颈迁移现象,揭示块大小、令牌保留率、注意力局部性与检索范围如何影响吞吐与精度权衡。在多个流式基准上,Qwen2.5-VL-3B/7B-Instruct实验表明,ViCoStream在单张A100 GPU上实现134 FPS视频吞吐,首次平均首字时间低于50毫秒,同时精度接近全历史基线。
原文摘要 · Abstract (English)
Streaming VideoLLMs must continuously process incoming video while maintaining low query latency, making both video-ingestion throughput and query-time responsiveness critical for real-time deployment. Existing methods largely focus on accelerating individual modules, such as visual encoding, token pruning, or KV-cache compression, but provide limited insight into whether the resulting system can sustain real-time streaming performance. We formulate streaming VideoLLM inference as a coordinated pipeline spanning visual preprocessing, visual encoding, token dropping, and LLM prefilling/decoding. Building on this formulation, we propose ViCoStream (Video Coordinated Streaming), a stage-wise coordinated streaming framework that combines chunk-wise execution, CUDA-stream overlap, visual token control, bounded visual attention, and query-side retrieval to bound per-chunk computation and memory costs. We further provide a systematic study of bottleneck migration, revealing how chunk size, token retention, attention locality, and retrieval scope shape the throughput-accuracy trade-off. Experiments with Qwen2.5-VL-3B/7B-Instruct across multiple streaming benchmarks show that ViCoStream achieves 134 FPS video throughput and less than 50 ms TTFT on a single A100 GPU while maintaining accuracy close to full-history baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。