早压缩视觉令牌,让视频大模型更快出结果。
EarlyTom: Early Token Compression Completes Fast Video Understanding

- 在视觉编码器内早期压缩令牌,减少计算负担。
- 单卡提速2.65倍,能耗降低61%,准确率不变。
- 适合需要低延迟的视频理解实时应用。
视频大语言模型(Video-LLMs)在视频理解任务中表现出色,但其实际部署受限于处理海量视觉令牌带来的效率问题。尽管近期方法在保持与全令牌基线相当精度的前提下实现了极低的令牌保留率,但多数仅在预填充后期进行压缩,未优化视觉编码器效率。本文首次表明,视觉编码占时至首字(TTFT)的很大比例。因此,应在编码器内部而非之后进行压缩,仍有较大探索空间。基于此,我们提出EarlyTom——一种无需训练的令牌压缩框架,在视觉编码器内实现早期令牌压缩,显著降低TTFT并提升吞吐量。此外,引入解耦的空间令牌选择策略,进一步提高压缩效果。在NVIDIA A100单卡上,EarlyTom使LLaVA-OneVision-7B模型的TTFT降低2.65倍,FLOPs减少61%,同时保持与全令牌基线相当的准确率,极大提升了视频大模型在真实场景中的实用性。
原文摘要 · Abstract (English)
Video large language models (Video-LLMs) have demonstrated strong capabilities in video understanding tasks. However, their practical deployment is still hindered by the inefficiency introduced by processing massive amounts of visual tokens. Although recent approaches achieve extremely low token retention ratios while maintaining accuracy comparable to full-token baselines, most of them perform compression only at the late stage of prefilling, leaving the efficiency of the vision encoder unoptimized. In this paper, we first show that vision encoding contributes a large portion to the time-to-first-token (TTFT). Therefore, instead of compressing visual tokens only after the vision encoder, performing compression inside the encoder still leaves substantial room for exploration. Based on this insight, we propose EarlyTom, a training-free token compression framework that performs early-stage visual token compression inside the vision encoder, enabling significantly better TTFT reduction and higher throughput. In addition, we introduce a decoupled spatial token selection strategy that improves the overall compression effectiveness. EarlyTom reduces TTFT by up to 2.65x and FLOPs by up to 61% on a single NVIDIA A100 GPU for the LLaVA-OneVision-7B model, while maintaining accuracy comparable to the full-token baseline. These improvements substantially enhance the practicality of deploying Video-LLMs in real-world production scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。