不训练即可加速视频大模型,仅用10%视觉标记保持99.1%性能
FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging
- 通过树状结构融合时空冗余,统一处理空间与时间相关性
- 仅保留10%视觉标记,仍保持LLaVA-OneVision 99.1%的性能
- 无需训练可直接部署,适合长视频推理和资源受限场景
尽管视频大语言模型(VLLMs)在视频理解方面表现出色,但需处理大量视觉标记,导致计算效率低下。现有加速框架通常独立压缩空间和时间冗余,忽略了时空关联,造成次优压缩。由于视频动态特性,高度相关的视觉特征随时间在位置、尺度、方向等属性上可能发生改变。基于此,我们提出FlashVID,一种无需训练的推理加速框架。具体地,FlashVID采用基于注意力与多样性的标记选择(ADTS)筛选基础视频表示中最具代表性标记,再通过树状时空标记合并(TSTM)实现细粒度的时空冗余消除。在三个代表性VLLM及五个视频理解基准上的实验表明方法有效且泛化性强。值得注意的是,仅保留10%视觉标记时,FlashVID在LLaVA-OneVision上保持99.1%性能;将其应用于Qwen2.5-VL,可在相同计算预算下将输入视频帧数提升10倍,相对性能提升8.6%。代码已开源。
原文摘要 · Abstract (English)
Although Video Large Language Models (VLLMs) have shown remarkable capabilities in video understanding, they are required to process high volumes of visual tokens, causing significant computational inefficiency. Existing VLLMs acceleration frameworks usually compress spatial and temporal redundancy independently, which overlooks the spatiotemporal relationships, thereby leading to suboptimal spatiotemporal compression. The highly correlated visual features are likely to change in spatial position, scale, orientation, and other attributes over time due to the dynamic nature of video. Building on this insight, we introduce FlashVID, a training-free inference acceleration framework for VLLMs. Specifically, FlashVID utilizes Attention and Diversity-based Token Selection (ADTS) to select the most representative tokens for basic video representation, then applies Tree-based Spatiotemporal Token Merging (TSTM) for fine-grained spatiotemporal redundancy elimination. Extensive experiments conducted on three representative VLLMs across five video understanding benchmarks demonstrate the effectiveness and generalization of our method. Notably, by retaining only 10% of visual tokens, FlashVID preserves 99.1% of the performance of LLaVA-OneVision. Consequently, FlashVID can serve as a training-free and plug-and-play module for extending long video frames, which enables a 10x increase in video frame input to Qwen2.5-VL, resulting in a relative improvement of 8.6% within the same computational budget. Code is available at https://github.com/Fanziyang-v/FlashVID.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。