通过分层压缩视频令牌,显著加速流式视频大模型推理。
Accelerating Streaming Video Large Language Models via Hierarchical Token Compression
- 设计两级压缩机制:缓存相似帧特征,剪枝冗余视觉令牌。
- 在保持99%准确率前提下,降低24.5%编码延迟和45.3%预填充延迟。
- 可无缝接入现有流式视频模型,适合实时视频理解场景。
流式视频大语言模型(VideoLLMs)在多项视频理解任务中表现优异,但因持续视频流中密集视觉令牌的高计算开销,在实时部署中面临挑战。主要瓶颈在于视觉变压器(ViT)编码阶段,重复处理时间上相似的帧导致效率低下;同时,大模型预填充阶段的令牌序列膨胀进一步加剧延迟与内存开销。为此,我们提出一种即插即用的分层框架STC,无缝集成至现有流式VideoLLMs,优化ViT编码与大模型预填充阶段。STC引入两个令牌级加速器:STC-Cacher通过缓存并复用时间上相似帧的特征,减少编码开销;STC-Pruner在令牌进入大模型前压缩序列,基于空间与时间相关性保留关键令牌。在四个基线流式VideoLLMs、五个基准测试上的实验表明,STC优于其他压缩方法。显著地,在ReKV框架上保留高达99%的准确率,同时将ViT编码延迟和大模型预填充延迟分别降低24.5%和45.3%。
原文摘要 · Abstract (English)
Streaming Video Large Language Models (VideoLLMs) have demonstrated impressive performance across various video understanding tasks, but they face significant challenges in real-time deployment due to the high computational cost of processing dense visual tokens from continuous video streams. In streaming video scenarios, the primary bottleneck lies in the Vision Transformer (ViT) encoding stage, where redundant processing of temporally similar frames leads to inefficiency. Additionally, inflated token sequences during LLM pre-filling further exacerbate latency and memory overhead. To address these challenges, we propose \textbf{S}treaming \textbf{T}oken \textbf{C}ompression (\textbf{STC}), a plug-and-play hierarchical framework that seamlessly integrates into existing streaming VideoLLMs, optimizing both ViT encoding and LLM pre-filling stages to accelerate processing. STC introduces two token-level accelerators: \textbf{STC-Cacher}, which reduces ViT encoding overhead by caching and reusing features from temporally similar frames, and \textbf{STC-Pruner}, which compresses the visual token sequence before it enters the LLM, preserving only the most salient tokens based on both spatial and temporal relevance. Extensive experiments on four baseline streaming VideoLLMs across five benchmarks demonstrate that STC outperforms other compression methods. Notably, STC retains up to \textbf{99\%} of accuracy on the ReKV framework while reducing ViT encoding latency and LLM pre-filling latency by \textbf{24.5\%} and \textbf{45.3\%}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。