利用视频编码器信号实现高效流式多模态推理，显著降低计算开销。
CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference
- 通过编码器元数据在线引导视觉与语言模型优化，无需离线训练。
- 吞吐量提升最高3倍，GPU计算减少87%，准确率仅降0~8% F1。
- 适合实时视频分析场景，尤其对低延迟、高并发服务有优势。
视频流分析是视觉语言模型服务的关键任务，但多模态推理成本高限制了可扩展性。现有系统通过利用视频流中的时空冗余来降低推理开销，但通常只针对视觉变压器（ViT）或大语言模型（LLM），缺乏端到端优化。此外，现有方法需依赖离线分析或高昂的在线计算来识别冗余，难以适应动态实时流。本文提出CodecSight，一种基于编码器信号的流式视频分析系统。核心观察是：视频编码器在压缩过程中已提取时序与空间结构作为副产品。CodecSight将此编码元数据作为低成本运行时信号，统一优化视频解码、视觉处理与LLM预填充阶段，直接操作压缩比特流天然带来传输减少。该系统实现编码前的视觉补丁剪枝和预填充期间的选择性键值缓存刷新，均为完全在线机制，无需离线训练。实验表明，CodecSight在吞吐量上相比最先进基线最高提升3倍，GPU计算消耗减少高达87%，同时保持竞争力的准确率，仅出现0~8%的F1分数下降。
原文摘要 · Abstract (English)
Video streaming analytics is a crucial workload for vision-language model serving, but the high cost of multimodal inference limits scalability. Prior systems reduce inference cost by exploiting temporal and spatial redundancy in video streams, but they target either the vision transformer (ViT) or the LLM with a limited view, leaving end-to-end opportunities untapped. Moreover, existing methods incur significant overhead to identify redundancy, either through offline profiling and training or costly online computation, making them ill-suited for dynamic real-time streams. We present CodecSight, a codec-guided streaming video analytics system, built on a key observation that video codecs already extract the temporal and spatial structure of each stream as a byproduct of compression. CodecSight treats this codec metadata as a low-cost runtime signal to unify optimization across video decoding, visual processing, and LLM prefilling, with transmission reduction as an inherent benefit of operating directly on compressed bitstreams. This drives codec-guided patch pruning before ViT encoding and selective key-value cache refresh during LLM prefilling, both of which are fully online and do not require offline training. Experiments show that CodecSight achieves an improvement in throughput of up to 3$\times$, and a reduction of up to 87% in GPU compute over state-of-the-art baselines, maintaining competitive accuracy with only 0$\sim$8% F1 drop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。