通过剪枝视频视觉标记，让大模型更快更省地理解视频内容。
PruneVid: Visual Token Pruning for Efficient Video Large Language Models
- 不需训练，通过合并时空标记减少冗余。
- 可剪掉80%以上视觉标记，性能仍保持领先。
- 适合追求高效视频理解的开发者和研究者。
本文提出PruneVid，一种用于提升多模态视频理解效率的视觉标记剪枝方法。大型语言模型（LLMs）因具备强大的视觉理解能力，在视频任务中表现优异，但视频数据中存在大量冗余，带来巨大计算负担。为此，我们设计了一种无需训练的方法：1）通过合并时空标记降低视频冗余；2）利用大模型的推理能力，选择性剪除与问题标记相关性低的视觉特征，从而提升效率。我们在多个视频基准上验证该方法，结果表明，PruneVid可在剪掉超过80%的视觉标记的同时，保持与多种模型架构兼容的竞争力表现，显著优于现有剪枝方法。代码已开源：https://github.com/Visual-AI/PruneVid。
原文摘要 · Abstract (English)
In this paper, we introduce PruneVid, a visual token pruning method designed to enhance the efficiency of multi-modal video understanding. Large Language Models (LLMs) have shown promising performance in video tasks due to their extended capabilities in comprehending visual modalities. However, the substantial redundancy in video data presents significant computational challenges for LLMs. To address this issue, we introduce a training-free method that 1) minimizes video redundancy by merging spatial-temporal tokens, and 2) leverages LLMs' reasoning capabilities to selectively prune visual features relevant to question tokens, enhancing model efficiency. We validate our method across multiple video benchmarks, which demonstrate that PruneVid can prune over 80% of tokens while maintaining competitive performance combined with different model networks. This highlights its superior effectiveness and efficiency compared to existing pruning methods. Code: https://github.com/Visual-AI/PruneVid.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。