动态压缩视频视觉令牌，大幅减少计算量且保持性能
DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding
- 按信息密度动态分组合并视觉令牌，实现自适应压缩
- 令牌数减至原大小的44.4%，在Video-MME和MLVU上分别达65.3%和72.5%
- 适合长视频处理，为高效视频建模提供新思路
主流视频建模方法（如LLava）将视频表示为视觉令牌序列，交由大语言模型处理。但该方式产生大量视觉令牌，尤其对长视频而言。本文提出一种新型动态视觉令牌压缩策略DynTok：自适应地将视觉令牌分组，并在组内合并，低信息密度区域实现高压缩，关键内容得以保留。该方法将令牌数量降至原始大小的44.4%，同时性能基本不变；在增加视频帧数时，进一步在Video-MME上达到65.3%、MLVU上达到72.5%的准确率。该方法揭示了视频令牌表示中的冗余性，为设计更高效的视频建模技术提供了新视角。
原文摘要 · Abstract (English)
Typical video modeling methods, such as LLava, represent videos as sequences of visual tokens, which are then processed by the LLM backbone for effective video understanding. However, this approach leads to a massive number of visual tokens, especially for long videos. A practical solution is to first extract relevant visual information from the large visual context before feeding it into the LLM backbone, thereby reducing computational overhead. In this work, we introduce DynTok, a novel \textbf{Dyn}amic video \textbf{Tok}en compression strategy. DynTok adaptively splits visual tokens into groups and merges them within each group, achieving high compression in regions with low information density while preserving essential content. Our method reduces the number of tokens to 44.4% of the original size while maintaining comparable performance. It further benefits from increasing the number of video frames and achieves 65.3% on Video-MME and 72.5% on MLVU. By applying this simple yet effective compression method, we expose the redundancy in video token representations and offer insights for designing more efficient video modeling techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。