仅保留2%视觉标记,仍保持90%性能,显著降低视频大模型计算开销。
Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention
- 将时空压缩重构为全局标记保留池中的分配问题,统一处理时空信息。
- 在低于2%保留率下仍保持90.1%基准性能,推理计算量降至约2.6%。
- 无需微调,可即插即用,适用于多种视频大模型架构。
视频大语言模型(Video-LLMs)因海量视觉标记面临高计算成本。现有压缩方法多采用分阶段时空分离策略,依赖特定阶段指标并隐含时空可分假设,在极低保留率下常导致分配失衡和关键视觉证据丢失。本文将标记压缩重新建模为全局标记保留池中的时空分配任务,提出一种统一选择机制,融合注意力权重与语义相似性,全局筛选贡献高、冗余低的标记。未选标记通过聚类合并并回填,保障信息完整性。在大模型内部引入文本感知合并,基于查询相关性进行二次压缩。本方法无需重训练,可作为即插即用模块适配现有Video-LLMs。实验表明,仅保留约2%视觉标记,即可在多个基准上维持90.1%的基线性能,同时将浮点运算量降至约2.6%。该优势在不同骨干网络间具有泛化能力,显著降低端到端推理延迟与内存消耗。所提统一时空标记压缩策略在超低标记保留条件下达到视频理解新基准。
原文摘要 · Abstract (English)
Video large language models (Video-LLMs) face high computational costs due to large volumes of visual tokens. Existing token compression methods typically adopt a two-stage spatiotemporal compression strategy, relying on stage-specific metrics and an implicit assumption of spatiotemporal separability. Under extremely low retention ratios, however, such approaches often result in unbalanced allocation and loss of visual evidence essential for question answering. We reformulate token compression as a spatiotemporal allocation task within a global token retention pool. We propose a unified selection mechanism that integrates attention weights and semantic similarity to globally select tokens with high contribution and low redundancy. Unselected tokens are merged via clustering and refilled, preserving information integrity. Inside the LLM, we further introduce text-aware merging to perform secondary compression based on query relevance. Without requiring retraining, our method serves as a plug-and-play module compatible with existing Video-LLMs. Experiments show that retaining only about 2% of visual tokens preserves 90.1% of baseline performance across multiple benchmarks, while reducing FLOPs to roughly 2.6%. These benefits generalize across diverse backbones, decreasing end-to-end inference latency and memory consumption. Our unified spatiotemporal token compression strategy establishes the state-of-the-art in video understanding under ultra-low token retention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。