根据视频内容动态分配帧数，提升生成效率
Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space
- 按内容自适应分配每帧令牌数，支持动态调整
- 在UCF-101和Kinetics-600上优于固定令牌方案
- 适合需要高效视频生成的场景，如流媒体
我们提出AdapTok，一种基于一维潜在空间的自适应时序因果视频分词方法，可根据视频内容灵活分配不同帧的令牌数量。该方法采用块级掩码策略，在训练中随机丢弃每块的尾部令牌，并引入块因果评分器，预测使用不同令牌数对视频帧重建质量的影响。推理时，基于整数线性规划的自适应分配策略，结合预测得分动态调整令牌使用。该设计实现了样本级、内容感知且时序动态的令牌分配，同时控制总体预算。在UCF-101与Kinetics-600上的视频重建与生成实验表明，无需额外图像数据，AdapTok在不同令牌预算下均持续提升重建质量与生成性能，推动更可扩展、高效的生成式视频建模。
原文摘要 · Abstract (English)
We propose AdapTok, an adaptive temporal causal video tokenizer that can flexibly allocate tokens for different frames based on video content. AdapTok is equipped with a block-wise masking strategy that randomly drops tail tokens of each block during training, and a block causal scorer to predict the reconstruction quality of video frames using different numbers of tokens. During inference, an adaptive token allocation strategy based on integer linear programming is further proposed to adjust token usage given predicted scores. Such design allows for sample-wise, content-aware, and temporally dynamic token allocation under a controllable overall budget. Extensive experiments for video reconstruction and generation on UCF-101 and Kinetics-600 demonstrate the effectiveness of our approach. Without additional image data, AdapTok consistently improves reconstruction quality and generation performance under different token budgets, allowing for more scalable and token-efficient generative video modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。