视频语言模型通过自适应分阶段剪枝,大幅降低计算量同时保持高精度。
Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models

- 分两阶段剪枝:先去冗余帧,再按内容自适应裁剪帧内令牌
- 在仅保留10%令牌时,视频描述任务准确率提升7%,计算量降低95%
- 无需训练即可部署,适合边缘设备和实时监控场景
视觉语言模型在图像与视频理解方面表现优异,但因每张图像需处理数千个令牌,导致推理延迟过高,限制了在资源受限的边缘设备及实时监控中的应用。这一问题在视频处理中尤为突出,需同时分析多帧画面。现有令牌压缩技术主要针对单图输入,未能利用视频序列中的时空冗余性。且多数方法采用固定剪枝比例,无法根据内容差异动态调整,易丢失关键信息。为此,本文提出一种专为视频设计的两阶段自适应令牌剪枝策略:第一阶段剔除冗余帧,第二阶段在保留帧内按内容自适应进行令牌剪枝。剪枝比例基于令牌嵌入的相关性结构量化冗余程度动态确定。该方法完全后置,无需额外训练或微调,在视频字幕基准测试中,仅保留10%令牌时准确率提升7%,计算量降低95%。
原文摘要 · Abstract (English)
Vision-language models excel at image and video understanding but suffer from high inference latency due to the need to process thousands of tokens per image, limiting their deployment on resource-constrained edge devices and in real-time surveillance applications. This challenge is further amplified in video processing, where multiple frames must be analyzed simultaneously. Existing token reduction techniques are largely developed for single-image inputs and therefore fail to account for the temporal and inter-frame redundancies present in video sequences. In addition, these methods generally rely on a fixed, uniform pruning ratio applied across all inputs, which is suboptimal because the degree of redundancy can vary significantly between different videos, necessitating content-dependent pruning levels to preserve critical information. To address these limitations, we propose a two-stage adaptive token pruning strategy specifically designed for video processing. In the first stage, we prune out the redundant frames, and in the second stage, token-level pruning is applied within the retained frames. Crucially, the pruning ratio in the second stage is determined adaptively based on the content of each video. This is achieved by analyzing the correlation structure of token embeddings to quantify redundancy, which is used to determine the ratio. Importantly, our method is entirely post-hoc and requires no additional training or fine-tuning, while achieving strong empirical gains; notably, it improves accuracy by +7\% on a video captioning benchmark at 10\% token retention, while reducing computation TFLOPs by 95\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。