用全局码本离线压缩视频令牌,一次学习全程复用,提速降耗。
Rethinking Video Token Compression with a Global Codebook: Learning Once, Compressing Everywhere

- 离线训练全局码本,线上通过查表聚合实现轻量压缩
- 在多个基准上保持竞争力,推理延迟最低
- 通用性强,无需为不同模型重设计压缩方案
视频大语言模型将视频表示为密集的视觉令牌序列,其长度随时间与空间范围增长。这些令牌常包含大量由重复视觉模式带来的冗余,导致后续语言模型处理时产生不必要的计算开销。现有令牌压缩方法(如剪枝、合并)在推理时在线执行,需对每段视频重复计算,且多依赖特定模型设计,通用性差。本文提出 extbf{ONCE},一种可插拔的视频令牌压缩框架,重新思考该范式:将昂贵的压缩过程移至离线阶段。通过在视觉特征空间中学习一个频率感知的全局码本,仅需一次训练即可长期复用;线上通过码本查找与聚合实现轻量级压缩,显著减少每视频的重复计算,并避免模型特异性设计。在多个视频理解基准上与多种压缩基线对比的实验证明,该方法在保持良好性能的同时,实现了最低的推理延迟,达成优异的精度-效率权衡。
原文摘要 · Abstract (English)
Video large language models (Video-LLMs) represent videos as dense sequences of visual tokens, whose length grows with the temporal and spatial extent of the input. These tokens often contain substantial redundancy arising from repeated visual patterns, leading to unnecessary computation in the subsequent language-model processing. Existing token compression methods, including pruning and merging, perform compression online during inference, repeatedly incurring additional computation for each input video and often relying on model-specific designs that limit their generality, we instead rethink this paradigm by shifting the costly compression process offline. We propose \textbf{ONCE}, a plug-in video token compression framework that introduces an offline-to-online paradigm: a frequency-aware global codebook is learned once in the visual feature space and reused for lightweight online compression through codebook lookup and aggregation, reducing repeated per-video computation and the need for model-specific compression designs. Extensive experiments across multiple video understanding benchmarks and against diverse compression baselines demonstrate that our approach achieves a strong accuracy-efficiency trade-off, maintaining competitive performance while achieving the lowest inference latency among compared methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。