不重新训练模型，通过合并视频令牌实现2.5倍加速且精度损失仅0.55%。
Video, How Do Your Tokens Merge?
- 采用无需训练的视频令牌合并方法，适配多种视觉变换器架构。
- 在三个数据集上实现约2.5倍推理速度提升，平均精度下降0.55%。
- 适用于需要高效视频理解的场景，尤其适合资源受限部署。
视频变换器模型因输入的时空规模大而需要大量计算资源。为应对这一挑战，近期方法提出对图像模型的令牌进行随机或学习式删除或合并。令牌合并具有诸多优势：可无缝集成至任意视觉变换器、无需模型重训练，并能将本会被丢弃的信息传递至模型深层。然而，此前尚未在时序复杂的视频理解数据集上系统评估过视频令牌合并的效果。本文针对四种视频变换器，在三个涵盖粗粒度与细粒度动作识别的数据集上开展无训练令牌合并的全面实验，探索最佳实践。结果表明，视频令牌合并可在实现约2.5倍加速的同时保持较高精度（以ViViT为例，平均精度仅下降0.55%）。代码已公开于 https://github.com/sjpollard/video-how-do-your-tokens-merge。
原文摘要 · Abstract (English)
Video transformer models require huge amounts of compute resources due to the spatio-temporal scaling of the input. Tackling this, recent methods have proposed to drop or merge tokens for image models, whether randomly or via learned methods. Merging tokens has many benefits: it can be plugged into any vision transformer, does not require model re-training, and it propagates information that would otherwise be dropped through the model. Before now, video token merging has not been evaluated on temporally complex datasets for video understanding. In this work, we explore training-free token merging for video to provide comprehensive experiments and find best practices across four video transformers on three datasets that exhibit coarse and fine-grained action recognition. Our results showcase the benefits of video token merging with a speedup of around $2.5$X while maintaining accuracy (avg. $-0.55\%$ for ViViT). Code available at https://github.com/sjpollard/video-how-do-your-tokens-merge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。