动态合并视频片段，提升长视频理解效率
Video Token Merging for Long-form Video Understanding
- 根据视频内容重要性动态合并token，避免信息丢失
- 在多个数据集上表现优于或接近基线模型
- 内存消耗降低84%，处理速度提升6.89倍
随着视频理解数据与模型规模的快速扩展，基于Transformer的模型处理长视频输入面临实际挑战。相较于采样或丢弃token可能造成的信息损失，令牌合并展现出良好前景。然而，将其应用于长视频并非易事。本文认为，令牌合并不应仅依赖视频块间的相似性，还应考虑其显著性。为此，我们探索了多种长视频分类中的令牌合并策略：从图像令牌合并的简单扩展，到区域集中式合并，最终提出可学习的视频令牌合并（VTM）算法，依据令牌显著性动态合并。大量实验表明，该方法在LVU、COIN和Breakfast数据集上表现优异，同时相较基线算法，内存占用降低84%，吞吐量提升约6.89倍。
原文摘要 · Abstract (English)
As the scale of data and models for video understanding rapidly expand, handling long-form video input in transformer-based models presents a practical challenge. Rather than resorting to input sampling or token dropping, which may result in information loss, token merging shows promising results when used in collaboration with transformers. However, the application of token merging for long-form video processing is not trivial. We begin with the premise that token merging should not rely solely on the similarity of video tokens; the saliency of tokens should also be considered. To address this, we explore various video token merging strategies for long-form video classification, starting with a simple extension of image token merging, moving to region-concentrated merging, and finally proposing a learnable video token merging (VTM) algorithm that dynamically merges tokens based on their saliency. Extensive experimental results show that we achieve better or comparable performances on the LVU, COIN, and Breakfast datasets. Moreover, our approach significantly reduces memory costs by 84% and boosts throughput by approximately 6.89 times compared to baseline algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。