通过时空森林建模,实现视频多模态大模型的高效高比例令牌压缩。
ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling
- 构建跨帧语义-空间-时间约束的令牌森林,全局理解视频内容。
- 保留95.8%准确率的同时减少90%令牌,优于现有方法。
- 无需训练,适合资源受限场景下的视频多模态推理。
由于显著降低计算与内存开销,令牌压缩已成为多模态大模型(MLLMs)的研究热点,并在图像-语言任务中取得显著进展。然而,在视频任务中,现有方法仍难以实现高比例令牌压缩。我们归因于对时序与连续视频内容建模不足,提出一种新颖且无需训练的视频MLLM令牌剪枝方法——ForestPrune,通过时空森林建模实现高效高比例剪枝。具体而言,ForestPrune基于语义、空间和时间约束在视频帧间构建令牌森林,实现对视频的整体理解;随后根据树深度与节点角色评估令牌树与节点重要性,获得全局最优剪枝决策。为验证有效性,我们在代表性视频MLLMs LLaVA-Video与LLaVA-OneVision上进行大量实验。结果表明:在保持95.8%平均准确率的同时,将令牌量减少90%(以LLaVA-OneVision为例);相比对比方法,其在MLVU上提升10.1%准确率,在LLaVA-Video上剪枝耗时减少81.4%(相较FrameFusion)。
原文摘要 · Abstract (English)
Due to the great saving of computation and memory overhead, token compression has become a research hot-spot for MLLMs and achieved remarkable progress in image-language tasks. However, for the video, existing methods still fall short of high-ratio token compression. We attribute this shortcoming to the insufficient modeling of temporal and continual video content, and propose a novel and training-free token pruning method for video MLLMs, termed ForestPrune, which achieves effective and high-ratio pruning via Spatial-temporal Forest Modeling. In practice, ForestPrune construct token forests across video frames based on the semantic, spatial and temporal constraints, making an overall comprehension of videos. Afterwards, ForestPrune evaluates the importance of token trees and nodes based on tree depth and node roles, thereby obtaining a globally optimal pruning decision. To validate ForestPrune, we apply it to two representative video MLLMs, namely LLaVA-Video and LLaVA-OneVision, and conduct extensive experiments on a bunch of video benchmarks. The experimental results not only show the great effectiveness for video MLLMs, e.g., retaining 95.8% average accuracy while reducing 90% tokens for LLaVA-OneVision, but also show its superior performance and efficiency than the compared token compression methods, e.g., +10.1% accuracy on MLVU and -81.4% pruning time than FrameFusion on LLaVA-Video.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。