用事件感知分阶段压缩视觉令牌，加速长视频理解
METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understanding
- 按事件重要性分三阶段压缩冗余视觉令牌
- 减少80.6%计算量，节省93.5%缓存内存
- 无需训练，适配现有视频大模型
视频大语言模型（VLLMs）在视频理解方面取得显著进展，但处理长视频仍面临高计算开销和视觉数据冗余的挑战。本文提出METok——一种无需训练的多阶段事件感知令牌压缩框架，通过三个关键阶段实现加速：(1) 视觉编码阶段基于事件感知的压缩；(2) 预填充阶段根据语义对齐与事件重要性进行层次化令牌剪枝；(3) 解码阶段优化KV缓存以进一步降低内存占用。在多个视频基准上的实验表明，METok能动态选择有信息量的视觉令牌，在保持甚至提升准确率的同时，实现效率与精度的最佳平衡。例如，将LongVA-7B与METok结合，可实现80.6%的FLOPs减少和93.5%的KV缓存内存节省。
原文摘要 · Abstract (English)
Recent advances in Video Large Language Models (VLLMs) have significantly enhanced their ability to understand video content. Nonetheless, processing long videos remains challenging due to high computational demands and the redundancy present in the visual data. In this work, we propose METok, a training-free, Multi-stage Event-based Token compression framework designed to accelerate VLLMs' inference while preserving accuracy. METok progressively eliminates redundant visual tokens across three critical stages: (1) event-aware compression during vision encoding, (2) hierarchical token pruning in the prefilling stage based on semantic alignment and event importance, and (3) a decoding-stage KV Cache optimization that further reduces memory consumption. Our experiments on diverse video benchmarks demonstrate that METok achieves an optimal trade-off between efficiency and accuracy by dynamically selecting informative visual tokens. For instance, equipping LongVA-7B with METok realizes an 80.6% FLOPs reduction and 93.5% KV Cache memory savings, all while maintaining comparable or even superior accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。