动态分配视频帧的标记数，让复杂内容用更多标记，简单内容用更少。
ElasticTok: Adaptive Tokenization for Image and Video
- 根据前一帧自适应决定当前帧的标记数量。
- 实验表明该方法显著提升标记使用效率。
- 适合需要处理长视频的多模态模型与世界模型开发者。
高效视频标记化仍是训练通用视觉模型的关键瓶颈，这类模型需处理长视频序列。现有方法将视频编码为固定数量的标记，标记过少导致信息严重损失，过多则造成序列过长。本文提出ElasticTok，通过依赖前一帧来自适应地将当前帧编码为可变数量的标记。为实现计算可扩展性，我们引入一种掩码技术，随机丢弃每帧标记编码末尾的若干标记。推理时，ElasticTok可根据需要动态分配标记：复杂数据使用更多标记，简单数据仅用少量标记。在图像和视频上的实证评估证明了该方法在标记使用效率方面的有效性，为未来更强大的多模态模型、世界模型和智能体的发展铺平道路。
原文摘要 · Abstract (English)
Efficient video tokenization remains a key bottleneck in learning general purpose vision models that are capable of processing long video sequences. Prevailing approaches are restricted to encoding videos to a fixed number of tokens, where too few tokens will result in overly lossy encodings, and too many tokens will result in prohibitively long sequence lengths. In this work, we introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens. To enable this in a computationally scalable way, we propose a masking technique that drops a random number of tokens at the end of each frames's token encoding. During inference, ElasticTok can dynamically allocate tokens when needed -- more complex data can leverage more tokens, while simpler data only needs a few tokens. Our empirical evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage, paving the way for future development of more powerful multimodal models, world models, and agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。