通过分层合并视频时空令牌，实现零训练加速。
Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs
- 基于四叉树结构分层生成空间令牌，再沿时间轴定向合并。
- 50%令牌预算下提速2倍，仅损失0.5%准确率；30%预算下提速3倍。
- 无需重训练，支持跨问题复用缓存，适合高效视频问答场景。
视频大语言模型（Video LLMs）通过大量时空令牌实现强大视频理解，但计算量随令牌数量呈二次增长。为此，我们提出一种无训练的时空令牌合并方法——STTM。核心思想是利用视频数据中被忽视的局部时空冗余。STTM首先通过四叉树结构从粗到细生成每帧的多粒度空间令牌，再在时间维度上执行定向成对合并。该分解式合并策略在六个视频问答基准上优于现有令牌压缩方法。值得注意的是，在50%令牌预算下，STTM实现2倍加速且仅损失0.5%准确率；在30%预算下提速3倍，准确率下降2%。此外，STTM为查询无关设计，支持同一视频不同问题间键值缓存复用。项目页面见 https://www.jshyun.me/projects/sttm。
原文摘要 · Abstract (English)
Video large language models (LLMs) achieve strong video understanding by leveraging a large number of spatio-temporal tokens, but suffer from quadratic computational scaling with token count. To address this, we propose a training-free spatio-temporal token merging method, named STTM. Our key insight is to exploit local spatial and temporal redundancy in video data which has been overlooked in prior work. STTM first transforms each frame into multi-granular spatial tokens using a coarse-to-fine search over a quadtree structure, then performs directed pairwise merging across the temporal dimension. This decomposed merging approach outperforms existing token reduction methods across six video QA benchmarks. Notably, STTM achieves a 2$\times$ speed-up with only a 0.5% accuracy drop under a 50% token budget, and a 3$\times$ speed-up with just a 2% drop under a 30% budget. Moreover, STTM is query-agnostic, allowing KV cache reuse across different questions for the same video. The project page is available at https://www.jshyun.me/projects/sttm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。