提出新型视频令牌压缩方法,实时降低大模型推理延迟。
Think in Sets for Streaming Video Token Compression

- 将令牌压缩视为集合选择,基于新增价值评估保留内容
- 在保持99.6%准确率的同时,预填充延迟降低46%
- 适合实时视频流处理场景,无需训练可直接部署
Streaming VideoLLMs 在处理视频帧时采用因果方式,视觉令牌持续增长,因此压缩对控制预填充延迟和内存至关重要。现有无训练方法独立排序令牌,忽略了保留令牌间的边际增益交互。我们提出应将流式视频令牌压缩建模为集合选择问题,每个候选令牌的价值取决于其相较于已有保留令牌带来的新增信息量。不同于面向离线任务的集合方法,流式处理需逐帧进行因果剪枝,因此建模跨帧交互需要显式的历史参考。这引发参考集困境:参考集需充分代表先前传达内容,同时保持容量有限以支持实时推理。我们提出 NovaCov,据我们所知首个面向流式视频的无训练、即插即用型集合式令牌压缩器。NovaCov 维护一个容量受限、按新近度加权的历史参考库,并优化双分支子模覆盖目标函数,既保留当前帧代表性内容,又优先保留历史未充分覆盖的信息。两个分支均为设施定位函数,因此贪婪选择仍能保证 (1-1/e) 近似率。在流式与离线基准上,NovaCov 均优于现有无训练压缩方法,在保留 ReKV 99.6% 准确率的同时,将 LLM 预填充延迟降低 46%。
原文摘要 · Abstract (English)
Streaming VideoLLMs process frames causally while visual tokens grow continuously, making compression essential for controlling prefilling latency and memory. Existing training-free methods independently rank tokens, ignoring marginal-gain interactions among retained tokens. We argue that streaming video token compression should instead be formulated as set selection, where each candidate is valued by what it adds beyond the tokens already retained. Unlike existing set-wise methods designed for offline tasks, streaming makes causal, frame-by-frame pruning decisions, so modeling cross-frame interactions requires an explicit historical reference. This creates a reference-set dilemma: the reference must adequately represent previously conveyed content while remaining bounded for real-time inference. We introduce NovaCov, to our knowledge the first training-free, plug-and-play set-wise token compressor designed for streaming video. NovaCov maintains a capacity-bounded, recency-weighted Historical Reference Bank and optimizes a dual-branch submodular coverage objective that preserves representative current-frame content while prioritizing information insufficiently covered by history. Both branches are facility-location functions, so greedy selection retains the classical (1-1/e) approximation guarantee. Across streaming and offline benchmarks, NovaCov outperforms existing training-free compression methods, retaining 99.6% of ReKV accuracy while reducing LLM prefilling latency by 46%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。