通过分层动态剪枝，用30%的视频令牌实现顶尖性能。
HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models
- 分三阶段剪枝：按片段、帧内相似性、模型层级逐步去冗余
- 仅保留30%令牌，仍保持98%以上原模型性能
- 适合需要高效部署视频大模型的研究者和开发者
视频大语言模型（VideoLLMs）在视频理解任务中表现优异，但海量输入视频令牌带来巨大计算负担。现有方法多在输入层面剪枝，忽略了视频与大语言模型内部的多模态信息结构。为此，本文提出HieraVid，一种分层动态剪枝框架，可渐进式减少视觉冗余。基于视频具有片段-帧结构、且多模态信息在模型中单向传播的观察，将剪枝分为三个层次：1）片段级，对视频进行时序分割并空间合并；2）帧级，在同一片段内联合剪除相似帧以保留多样性；3）层级，随着模型层数增加，冗余逐渐减少而不影响性能。我们在四个常用视频理解基准上进行了广泛实验。令人瞩目的是，仅保留30%令牌时，HieraVid达到新最优性能，同时保持了LLaVA-Video-7B和LLaVA-OneVision-7B超过98%和99%的性能水平。
原文摘要 · Abstract (English)
Video Large Language Models (VideoLLMs) have demonstrated impressive capabilities in video understanding, yet the massive number of input video tokens incurs a significant computational burden for deployment. Existing methods mainly prune video tokens at input level while neglecting the inherent information structure embedded in videos and large language models (LLMs). To address this, we propose HieraVid, a hierarchical pruning framework that progressively and dynamically reduces visual redundancy. Based on two observations that videos possess the segment-frame structure and LLMs internally propagate multi-modal information unidirectionally, we decompose pruning into three levels: 1) segment-level, where video tokens are first temporally segmented and spatially merged; 2) frame-level, where similar frames within the same segment are jointly pruned to preserve diversity; 3) layer-level, redundancy gradually shrinks as LLM layer increases w/o compromising performance. We conduct extensive experiments on four widely used video understanding benchmarks to comprehensively evaluate the effectiveness of HieraVid. Remarkably, with only 30% of tokens retained, HieraVid achieves new state-of-the-art performance, while maintaining over 98% and 99% of the performance of LLaVA-Video-7B and LLaVA-OneVision-7B, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。