提出新方法合并视频令牌,让大模型更快更省资源。
HoliTom: Holistic Token Merging for Fast Video Large Language Models
- 内外层结合:先全局分段再空间时间合并,大幅减少冗余令牌
- 压缩超90%视觉令牌,计算量降至原6.9%,性能保留99.1%
- 无需训练,适配主流视频大模型,显著提升推理速度
视频大语言模型(video LLMs)在视频理解上表现优异,但因冗余视频令牌导致计算效率低下。现有剪枝方法中,基于LLM内部的剪枝(如FastV)在浅层引入额外开销;而外部剪枝仅处理单帧或短时窗的空间冗余,忽视长视频序列中的全局时序动态与关联,难以充分挖掘视频可压缩性。本文提出HoliTom——一种无需训练的全局化令牌合并框架,通过全局感知的时序分段实现外层剪枝,再进行时空合并,使视觉令牌减少超90%,显著降低LLM负担。同时设计基于内层令牌相似性的合并策略,与外层剪枝兼容。在LLaVA-OneVision-7B上评估显示,该方法将计算成本降至原FLOPs的6.9%,性能保持99.1%;同时实现2.28倍的首次生成时间(TTFT)降低与1.32倍的解码吞吐加速,验证了集成剪枝的有效性与实用性。
原文摘要 · Abstract (English)
Video large language models (video LLMs) excel at video comprehension but face significant computational inefficiency due to redundant video tokens. Existing token pruning methods offer solutions. However, approaches operating within the LLM (inner-LLM pruning), such as FastV, incur intrinsic computational overhead in shallow layers. In contrast, methods performing token pruning before the LLM (outer-LLM pruning) primarily address spatial redundancy within individual frames or limited temporal windows, neglecting the crucial global temporal dynamics and correlations across longer video sequences. This leads to sub-optimal spatio-temporal reduction and does not leverage video compressibility fully. Crucially, the synergistic potential and mutual influence of combining these strategies remain unexplored. To further reduce redundancy, we introduce HoliTom, a novel training-free holistic token merging framework. HoliTom employs outer-LLM pruning through global redundancy-aware temporal segmentation, followed by spatial-temporal merging to reduce visual tokens by over 90%, significantly alleviating the LLM's computational burden. Complementing this, we introduce a robust inner-LLM token similarity-based merging approach, designed for superior performance and compatibility with outer-LLM pruning. Evaluations demonstrate our method's promising efficiency-performance trade-off on LLaVA-OneVision-7B, reducing computational costs to 6.9% of FLOPs while maintaining 99.1% of the original performance. Furthermore, we achieve a 2.28x reduction in Time-To-First-Token (TTFT) and a 1.32x acceleration in decoding throughput, highlighting the practical benefits of our integrated pruning approach for efficient video LLMs inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。