通过全局时空密度分析,高效剪枝视频模型冗余视觉标记。
GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Models

- 将视频视为连续时空流,综合时序与空间密度建模。
- 剪掉75%视觉标记,性能损失不足0.2%,推理提速1.17倍。
- 适用于各类视频大模型,无需修改结构,通用性强。
视频大语言模型(VideoLLMs)虽具备强大视频理解能力,但长视频推理成本高昂,源于大量冗余的时空视觉标记。现有剪枝方法多依赖分段局部剪枝,将视频分割为独立片段并分别筛选标记,易忽略短而语义密集片段,且可能丢弃局部不显著但全局关键的标记。为此,本文提出GSTEP(全局时空密度剪枝)框架,将视频建模为连续的时空信息流。GSTEP通过融合平滑后的帧级变化信号获得连续时序密度,结合帧内空间密度,构建标记级时空密度,并通过全局采样平衡信息密度与覆盖范围。在多个VideoLLMs和公开基准上的实验证明,GSTEP在不同模型架构与评估设置下均实现优异的准确率-效率权衡。以LLaVA-OneVision-7B为例,该方法剪除75%视觉标记,跨基准平均性能保持率达100.2%,端到端推理速度提升1.17倍。
原文摘要 · Abstract (English)
Video large language models (VideoLLMs) achieve strong video understanding performance, but their inference remains expensive due to the large number of redundant spatio-temporal visual tokens in long videos. Existing token pruning methods alleviate this cost by reducing redundant tokens, yet most of them rely on segment-level local pruning, where videos are partitioned into isolated segments and tokens are selected independently within each segment. Such designs may under-preserve short but semantically dense segments and discard tokens that appear non-salient locally but remain critical from a global perspective. To address this issue, we propose GSTEP (Global Spatio-Temporal Density Pruning), a plug-and-play pruning framework that models video as a continuous spatio-temporal information flow. GSTEP constructs a token-level spatio-temporal density by combining a continuous temporal density, obtained from a smoothed centered frame-level change signal, with intra-frame spatial density, and then performs global token sampling by jointly balancing information density and coverage. Extensive experiments on multiple VideoLLMs and public benchmarks demonstrate that GSTEP consistently achieves strong accuracy-efficiency trade-offs and generalizes well across model architectures and evaluation settings. On LLaVA-OneVision-7B, GSTEP prunes 75% of visual tokens, preserves up to 100.2% of the original average performance across benchmarks, and achieves a 1.17 end-to-end speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。