通过剪枝静止帧块,显著加速视频语言模型推理。
Efficient Video Sampling: Pruning Temporally Redundant Tokens for Faster VLM Inference
- 识别连续帧中不变的视觉区域并移除冗余信息
- 推理速度提升4倍,准确率损失极小
- 无需重训练,适合各类视频理解任务
视觉-语言模型(VLM)正从静态图像理解扩展到视频推理,但其可扩展性受限于密集帧序列处理带来的二次方计算开销。长视频常超出现代语言模型的令牌预算,导致上下文限制和延迟问题。本文提出高效视频采样(EVS),一种简单、即插即用的方法,通过识别并剪枝时间上静止的图像块——在连续帧间保持不变的空间区域——来减少令牌冗余。EVS保持位置标识,无需架构改动或重新训练。实验表明,EVS显著降低令牌数量,同时保持语义保真度,实现更快推理与更长输入序列。在推理阶段应用时,可使大语言模型(LLM)首次响应时间(TTFT)缩短最高达4倍,且准确率损失微小。结合随机剪枝率的微调阶段后,模型对不同压缩水平具备鲁棒性,在激进剪枝下仍保持完整性能。大量实验验证了EVS在效率-准确率权衡上的持续优势,实现了无损质量的可扩展视频-语言理解。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have recently expanded from static image understanding to video reasoning, but their scalability is fundamentally limited by the quadratic cost of processing dense frame sequences. Long videos often exceed the token budget of modern language models, leading to severe context limitations and latency issues. We introduce Efficient Video Sampling (EVS), a simple, plug-and-play method for reducing token redundancy in videos by identifying and pruning temporally static patches -- spatial regions that remain unchanged across consecutive frames. EVS preserves positional identity, requires no architectural changes or retraining. We show that EVS substantially reduces token count while maintaining semantic fidelity, enabling faster inference and longer input sequences. Applied at inference time, EVS reduces large language model (LLM) time-to-first-token (TTFT) by up to 4x with minimal accuracy loss. When combined with an uptraining phase using stochastic pruning rates, EVS yields models that are robust to varying compression levels and retain full performance under aggressive pruning. Extensive experiments demonstrate that EVS consistently improves efficiency-accuracy trade-offs, unlocking scalable video-language understanding without sacrificing quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。