arXiv:2503.11187cs.CV2025-03NeurIPS被引 48

通过动态密度剪枝,大幅减少视频大模型推理冗余,提速7倍且保持98%准确率。

FastVID: Dynamic Density Pruning for Fast Video Large Language Models

  • 按时间顺序分段并基于密度动态剪枝视频令牌
  • 在保留时空结构前提下,剪掉90.3%令牌,计算量降至8.3%
  • 适用于长/短视频任务,适合部署资源受限的视频大模型

视频大语言模型虽具备强大视频理解能力,但其实际部署受制于冗余视频令牌带来的高推理成本。现有剪枝方法未能有效利用视频数据中的时空冗余。为此,本文从时间上下文与视觉上下文两方面系统分析视频冗余,并提出面向快速视频大模型的动态密度剪枝方法FastVID。FastVID将视频动态划分为时序有序的片段,采用基于密度的令牌剪枝策略,以保留关键时空信息。实验表明,FastVID在多个短/长视频基准上均达到领先性能,覆盖LLaVA-OneVision、LLaVA-Video、Qwen2-VL及Qwen2.5-VL等主流模型。在LLaVA-OneVision-7B上,成功剪除90.3%视频令牌,将浮点运算量降至8.3%,预填充阶段加速7.1倍,同时保持98.0%原始准确率。代码已开源。

原文摘要 · Abstract (English)

Video Large Language Models have demonstrated strong video understanding capabilities, yet their practical deployment is hindered by substantial inference costs caused by redundant video tokens. Existing pruning techniques fail to effectively exploit the spatiotemporal redundancy present in video data. To bridge this gap, we perform a systematic analysis of video redundancy from two perspectives: temporal context and visual context. Leveraging these insights, we propose Dynamic Density Pruning for Fast Video LLMs termed FastVID. Specifically, FastVID dynamically partitions videos into temporally ordered segments to preserve temporal structure and applies a density-based token pruning strategy to maintain essential spatial and temporal information. Our method significantly reduces computational overhead while maintaining temporal and visual integrity. Extensive evaluations show that FastVID achieves state-of-the-art performance across various short- and long-video benchmarks on leading Video LLMs, including LLaVA-OneVision, LLaVA-Video, Qwen2-VL, and Qwen2.5-VL. Notably, on LLaVA-OneVision-7B, FastVID effectively prunes $\textbf{90.3%}$ of video tokens, reduces FLOPs to $\textbf{8.3%}$, and accelerates the LLM prefill stage by $\textbf{7.1}\times$, while maintaining $\textbf{98.0%}$ of the original accuracy. The code is available at https://github.com/LunarShen/FastVID.

视频生成模型压缩剪枝

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。