通过融合相似性与重要性,大幅减少视频模型的视觉标记数。
FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Models
- 按相邻帧对应标记计算相似性,合并冗余内容。
- 减少70%视觉标记,速度提升1.6至3.6倍,性能损失小于3%。
- 适合长视频处理的大型视觉语言模型使用。
随着对长视频和高分辨率视频处理需求的增长,大型视觉语言模型(LVLM)因海量视觉标记面临巨大负担。现有标记缩减方法主要基于重要性度量(如累积注意力得分)进行剪枝,但即使重要标记也可能因相邻帧间相似性及重复视觉元素而产生冗余。为此,本文提出FrameFusion,一种结合相似性合并与重要性剪枝的新型标记缩减方法。我们深入研究了标记相似性特征,发现三个关键规律:(1) 相邻帧中空间对应标记的余弦相似度高于其他标记对;(2) 高相似度在深层网络中显著下降;(3) 标记相似度排名在不同层间高度一致。基于此,FrameFusion仅在相邻帧对应标记间计算相似性,初期层执行标记合并,深层进行剪枝,并采用级联合并策略进一步提升效率。我们在六种不同规模的LVLM(2B至72B参数)上,使用五个涵盖视频检索、问答和时空理解任务的视频基准进行了全面评估。实验表明,FrameFusion可减少70%视觉标记,实现1.6–3.6倍端到端加速,平均性能损失低于3%。代码已开源:https://github.com/thu-nics/FrameFusion。
原文摘要 · Abstract (English)
The increasing demand to process long and high-resolution videos significantly burdens Large Vision-Language Models (LVLMs) due to the enormous number of visual tokens. Existing token reduction methods primarily prune tokens based on importance metrics, such as cumulative attention scores. However, even important tokens may exhibit high redundancy caused by similarity among adjacent video frames and repetitive visual elements. To address this limitation, we propose FrameFusion, a novel token reduction approach integrating similarity-based merging with importance-based pruning. We conduct a thorough study on token similarity characteristics, revealing three key insights: (1) spatially corresponding visual tokens between adjacent frames have higher cosine similarities compared to other token pairs; (2) high token similarities prominently decrease in deeper model layers; and (3) token similarity rankings are highly consistent across different layers. Guided by these observations, FrameFusion computes token similarities exclusively between corresponding visual tokens from adjacent frames, applies token merging at initial successive layers followed by pruning in deeper layers, and adopts a cascaded merging strategy to further enhance efficiency. We evaluate FrameFusion comprehensively across six diverse LVLMs, ranging from 2B to 72B parameters, using five video benchmarks encompassing video retrieval, question-answering, and spatial-temporal understanding tasks. Experiments show that FrameFusion reduces visual tokens by 70%, achieving 1.6-3.6x end-to-end speedups, with an average performance impact of less than 3%. Our code is available at: https://github.com/thu-nics/FrameFusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。