通过感知信息量压缩视频令牌,显著提升视频大模型效率。
InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models

- 用段级时序指纹差评估时间冗余,更抗噪声。
- 按内容独特性和熵值动态分配令牌预算,突出关键片段。
- 无需训练,在多个模型上实现85%令牌减少与4.24倍加速。
视频大语言模型在视频理解上表现强劲,但过多的视觉令牌带来巨大计算开销。现有无训练压缩方法依赖相邻帧相似性估计时间冗余,或按段长分配令牌,对帧级噪声敏感,难以捕捉真实视频中非均匀的信息分布。为此,我们提出InfoMerge,一种无训练的视觉令牌压缩方法,通过鲁棒的冗余估计和内容感知的预算分配提升令牌利用率。具体地,提出时序指纹差:一种段级二阶时间冗余估计策略,建模同一空间位置在各段内的时序相似结构;进一步引入内容感知预算分配(CABA),基于段落独特性和基于谱熵的表征丰富度动态分配段级令牌预算。通过减少重复保留冗余静态区域并增加对高信息量段的令牌投入,InfoMerge在有限令牌预算下实现更优性能。大量实验表明,InfoMerge在多个基准和骨干网络上均取得优异的效率-精度权衡,尤其在激进压缩下优势更明显。在LLaVA-OneVision-7B上,仅保留98.8%的平均性能,视觉令牌减少85%,预填充阶段提速4.24倍。
原文摘要 · Abstract (English)
Video Large Language Models (Video-LLMs) achieve strong performance in video understanding, but their excessive visual tokens bring substantial computational overhead. Existing training-free compression methods improve inference efficiency by reducing visual tokens, yet they often rely on local adjacent-frame similarity for temporal redundancy estimation or allocate token budgets mainly according to segment length. Such designs are sensitive to frame-level noise and fail to capture the non-uniform information distribution of real-world videos. To address these challenges, we propose InfoMerge, a training-free visual token compression method that improves token utilization through robust redundancy estimation and content-aware budget allocation. Specifically, we propose the Temporal Fingerprint Difference: a segment-level second-order temporal redundancy estimation strategy, which models the temporal similarity structure of tokens at the same spatial positions within each segment. We further introduce Content-Aware Budget Allocation (CABA), which dynamically allocates segment-level token budgets based on segment uniqueness and spectral-entropy-based representational richness. By reducing repeated preservation of redundant static regions and allocating more tokens to informative segments, InfoMerge makes better use of the limited token budget while maintaining strong performance. Extensive experiments show that InfoMerge achieves strong efficiency--accuracy trade-offs across multiple benchmarks and backbones, with more pronounced advantages under aggressive compression. On LLaVA-OneVision-7B, InfoMerge retains 98.8\% of the original average performance while reducing 85\% of visual tokens and achieving a 4.24-fold speedup in the prefill stage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。