用压缩思想去除视频重复帧，让Transformer训练快30%且不丢精度。
Don't Look Twice: Faster Video Transformers with Run-Length Tokenization
- 通过运行长度编码思想识别并合并重复视频块，减少输入令牌数。
- 训练加速超30%，长视频数据最多减少80%令牌量，准确率几乎不变。
- 无需调参、无额外计算开销，适合所有视频数据集快速部署。
视频Transformer训练慢源于输入令牌数量巨大，尽管许多视频片段在时间上重复。现有去重方法要么开销大，要么需针对不同数据集调参。本文提出运行长度令牌化（RLT），受数据压缩中运行长度编码启发，可在推理前高效识别并合并时间上重复的图像块，用单个块和位置编码替代，实现内容自适应去重。该方法无需调参、开销极小。实验表明，RLT使视频Transformer微调耗时减少30%，在30 FPS视频上训练提速超100%，长视频数据令牌数最多减少80%；无需训练即可提升模型吞吐量35%，准确率仅下降0.1%。
原文摘要 · Abstract (English)
Transformers are slow to train on videos due to extremely large numbers of input tokens, even though many video tokens are repeated over time. Existing methods to remove such uninformative tokens either have significant overhead, negating any speedup, or require tuning for different datasets and examples. We present Run-Length Tokenization (RLT), a simple approach to speed up video transformers inspired by run-length encoding for data compression. RLT efficiently finds and removes runs of patches that are repeated over time prior to model inference, then replaces them with a single patch and a positional encoding to represent the resulting token's new length. Our method is content-aware, requiring no tuning for different datasets, and fast, incurring negligible overhead. RLT yields a large speedup in training, reducing the wall-clock time to fine-tune a video transformer by 30% while matching baseline model performance. RLT also works without any training, increasing model throughput by 35% with only 0.1% drop in accuracy. RLT speeds up training at 30 FPS by more than 100%, and on longer video datasets, can reduce the token count by up to 80%. Our project page is at https://rccchoudhury.github.io/projects/rlt/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。