提出可变帧率视频分词器,提升生成效率与质量
VFRTok: Variable Frame Rates Video Tokenizer with Duration-Proportional Information Assumption
- 基于时长比例信息假设,实现非均匀帧率编码
- 仅用1/8令牌达领先生成保真度,重建质量优秀
- 适合需要高效视频生成的科研与工业场景
基于潜在扩散模型的现代视频生成框架因帧数比例信息假设存在分词效率问题。现有分词器采用固定时间压缩率,导致扩散模型计算成本随帧率线性增长。本文提出时长比例信息假设:视频信息容量上限与持续时间成正比,而非帧数。基于此,提出VFRTok——一种基于Transformer的视频分词器,通过编码器与解码器异步帧率训练实现可变帧率编解码。此外,提出部分旋转位置嵌入(Partial RoPE),解耦位置与内容建模,将相关补丁聚类为统一令牌。该方法显著提升内容感知能力,增强生成性能。得益于紧凑连续的时空表示,VFRTok在仅使用1/8令牌的情况下,实现了竞争性重建质量与最先进生成保真度。代码与权重已开源:https://github.com/KwaiVGI/VFRTok。
原文摘要 · Abstract (English)
Modern video generation frameworks based on Latent Diffusion Models suffer from inefficiencies in tokenization due to the Frame-Proportional Information Assumption. Existing tokenizers provide fixed temporal compression rates, causing the computational cost of the diffusion model to scale linearly with the frame rate. The paper proposes the Duration-Proportional Information Assumption: the upper bound on the information capacity of a video is proportional to the duration rather than the number of frames. Based on this insight, the paper introduces VFRTok, a Transformer-based video tokenizer, that enables variable frame rate encoding and decoding through asymmetric frame rate training between the encoder and decoder. Furthermore, the paper proposes Partial Rotary Position Embeddings (RoPE) to decouple position and content modeling, which groups correlated patches into unified tokens. The Partial RoPE effectively improves content-awareness, enhancing the video generation capability. Benefiting from the compact and continuous spatio-temporal representation, VFRTok achieves competitive reconstruction quality and state-of-the-art generation fidelity while using only 1/8 tokens compared to existing tokenizers. The code and weights are released at: https://github.com/KwaiVGI/VFRTok.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。