提出视觉令牌5大规律，实现高效视频理解
Principles of Visual Tokens for Efficient Video Understanding
- 发现视觉令牌价值呈帕累托分布，少数关键令牌承载主要信息
- 提出轻量模型LITE，在计算量与准确率平衡上超越主流方法
- 无需重训练即可跨数据集和任务泛化，适合实际部署
近年来，视频理解得益于Transformer架构的突破，但其计算开销大且视频数据冗余严重，效率问题日益突出。尽管已有方法通过令牌选择与合并提升效率，却普遍无法超越随机丢弃令牌的基线表现。本文深入分析该现象，揭示视觉令牌的五项本质规律：例如，令牌价值呈现明显帕累托分布——多数令牌价值极低，仅少数承载主要感知信息。基于此，我们提出轻量级视频模型LITE，能高效筛选少量关键令牌，在Kinetics-400与Something-Something-V2数据集上，在计算量（GFLOPs）与准确率的权衡中优于当前最优方法。实验还表明，LITE具备良好的跨数据集及跨任务泛化能力，无需重新训练。
原文摘要 · Abstract (English)
Video understanding has made huge strides in recent years, relying largely on the power of transformers. As this architecture is notoriously expensive and video data is highly redundant, research into improving efficiency has become particularly relevant. Some creative solutions include token selection and merging. While most methods succeed in reducing the cost of the model and maintaining accuracy, an interesting pattern arises: most methods do not outperform the baseline of randomly discarding tokens. In this paper we take a closer look at this phenomenon and observe 5 principles of the nature of visual tokens. For example, we observe that the value of tokens follows a clear Pareto-distribution where most tokens have remarkably low value, and just a few carry most of the perceptual information. We build on these and further insights to propose a lightweight video model, LITE, that can select a small number of tokens effectively, outperforming state-of-the-art and existing baselines across datasets (Kinetics-400 and Something-Something-V2) in the challenging trade-off of computation (GFLOPs) vs accuracy. Experiments also show that LITE generalizes across datasets and even other tasks without the need for retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。