根据视频内容动态调整分块数量,提升生成效率与质量
EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation
- 按视频复杂度自适应分配令牌数,避免冗余或不足
- 相比最佳基线减少24.4%令牌用量,且重建质量更优
- 适合追求高效视频生成的开发者和研究者
自回归视频生成模型依赖于将像素压缩为离散令牌序列的视频分词器。令牌序列长度在重建质量与下游生成计算成本之间起关键作用。传统分词器对不同视频的时间块采用统一令牌分配,常在简单、静态或重复片段上浪费令牌,而对动态或复杂片段支持不足。为此,我们提出EVATok框架,实现高效视频自适应分词。该框架为每段视频估计最优令牌分配以达到最佳质量-成本平衡,设计轻量级路由模块快速预测最优分配,并训练基于预测结果编码视频的自适应分词器。实验表明,EVATok显著提升视频重建与下游自回归生成的效率与整体质量。通过引入视频语义编码器的先进训练策略,其在UCF-101数据集上实现更优的重建效果与最先进的类别到视频生成性能,平均令牌使用量较之前最优方法LARP及固定长度基线至少减少24.4%。
原文摘要 · Abstract (English)
Autoregressive (AR) video generative models rely on video tokenizers that compress pixels into discrete token sequences. The length of these token sequences is crucial for balancing reconstruction quality against downstream generation computational cost. Traditional video tokenizers apply a uniform token assignment across temporal blocks of different videos, often wasting tokens on simple, static, or repetitive segments while underserving dynamic or complex ones. To address this inefficiency, we introduce $\textbf{EVATok}$, a framework to produce $\textbf{E}$fficient $\textbf{V}$ideo $\textbf{A}$daptive $\textbf{Tok}$enizers. Our framework estimates optimal token assignments for each video to achieve the best quality-cost trade-off, develops lightweight routers for fast prediction of these optimal assignments, and trains adaptive tokenizers that encode videos based on the assignments predicted by routers. We demonstrate that EVATok delivers substantial improvements in efficiency and overall quality for video reconstruction and downstream AR generation. Enhanced by our advanced training recipe that integrates video semantic encoders, EVATok achieves superior reconstruction and state-of-the-art class-to-video generation on UCF-101, with at least 24.4% savings in average token usage compared to the prior state-of-the-art LARP and our fixed-length baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。