arXiv:2603.18004cs.CVcs.AI2026-03被引 1

提出统一时空令牌评分方法,高效剪枝视频视觉语言模型中冗余信息。

Unified Spatio-Temporal Token Scoring for Efficient Video VLMs

  • 基于辅助损失和下游梯度,无须文本条件实现跨视觉与语言模块的统一剪枝。
  • 剪掉50%视觉令牌,训练推理效率提升62%,性能仅降0.7%。
  • 适用于长视频问答,支持测试时扩展,适合资源受限场景部署。

令牌剪枝对提升视觉语言模型(VLMs)的计算效率至关重要,尤其在存在时间冗余的视频任务中。现有方法通常仅在视觉变换器(ViT)内剪枝,或仅在大语言模型(LLM)中剪枝,且多依赖复杂文本条件机制。本文提出时空令牌评分(STTS),一种轻量级模块,可在不依赖文本条件或令牌合并的情况下,跨ViT与LLM统一剪枝视觉令牌,并支持端到端训练。通过辅助损失学习时间评分,利用LLM下游梯度学习空间评分,并结合高效打包算法,STTS可剪除全架构中50%的视觉令牌,在训练与推理阶段效率提升62%,13项短/长视频问答任务平均性能仅下降0.7%。采样帧数越多,效率增益越显著;对长视频问答应用测试时缩放,性能进一步提升0.5-1%。整体上,STTS是一种新颖、简单而高效的统一视觉令牌剪枝技术。

原文摘要 · Abstract (English)

Token pruning is essential for enhancing the computational efficiency of vision-language models (VLMs), particularly for video-based tasks where temporal redundancy is prevalent. Prior approaches typically prune tokens either (1) within the vision transformer (ViT) exclusively for unimodal perception tasks such as action recognition and object segmentation, without adapting to downstream vision-language tasks; or (2) only within the LLM while leaving the ViT output intact, often requiring complex text-conditioned token selection mechanisms. In this paper, we introduce Spatio-Temporal Token Scoring (STTS), a simple and lightweight module that prunes vision tokens across both the ViT and the LLM without text conditioning or token merging, and is fully compatible with end-to-end training. By learning how to score temporally via an auxiliary loss and spatially via LLM downstream gradients, aided by our efficient packing algorithm, STTS prunes 50% of vision tokens throughout the entire architecture, resulting in a 62% improvement in efficiency during both training and inference with only a 0.7% drop in average performance across 13 short and long video QA tasks. Efficiency gains increase with more sampled frames per video. Applying test-time scaling for long-video QA further yields performance gains of 0.5-1% compared to the baseline. Overall, STTS represents a novel, simple yet effective technique for unified, architecture-wide vision token pruning.

视频理解模型压缩视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。