通过动态选token提升视频模型训练效率，节省近90%计算量仍保持顶尖性能
Make Your Training Flexible: Towards Deployment-Efficient Video Models
- 引入可变采样网格与令牌选择机制，灵活优化输入信息
- 仅用1/4令牌即达以往顶级模型性能，计算节省近90%
- 兼容主流训练框架，适合部署受限场景的视频模型应用
现有视频训练方法通常在预设时空网格上固定采样，因视频冗余导致精度与计算效率不匹配，且难以适应下游任务的异构算力需求。为此，我们提出一种新测试设置——令牌优化（Token Optimization），旨在有限计算预算下最大化输入信息。通过灵活采样网格并结合令牌选择，我们设计了新型增强工具Flux，可无缝集成至多数主流视频训练框架中，几乎无额外成本显著提升模型鲁棒性。将Flux应用于大规模视频预训练后，得到的FluxViT在多种任务上达成新SOTA，且在仅使用1/4令牌时，性能仍可媲美以往最先进模型，实现近90%的计算节省。所有模型与数据已公开于https://github.com/OpenGVLab/FluxViT。
原文摘要 · Abstract (English)
Popular video training methods mainly operate on a fixed number of tokens sampled from a predetermined spatiotemporal grid, resulting in sub-optimal accuracy-computation trade-offs due to inherent video redundancy. They also lack adaptability to varying computational budgets for downstream tasks, hindering applications of the most competitive model in real-world scenes. We thus propose a new test setting, Token Optimization, for maximized input information across budgets, which optimizes the size-limited set of input tokens through token selection from more suitably sampled videos. To this end, we propose a novel augmentation tool termed Flux. By making the sampling grid flexible and leveraging token selection, it is easily adopted in most popular video training frameworks, boosting model robustness with nearly no additional cost. We integrate Flux in large-scale video pre-training, and the resulting FluxViT establishes new state-of-the-art results across extensive tasks at standard costs. Notably, with 1/4 tokens only, it can still match the performance of previous state-of-the-art models with Token Optimization, yielding nearly 90\% savings. All models and data are available at https://github.com/OpenGVLab/FluxViT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。