arXiv:2604.12887cs.CVcs.LG2026-04被引 4

视频分阶段生成细粒度令牌,更高效且支持长视频生成。

VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

论文配图:VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization
图 1 · 摘自论文原文
  • 视频用粗到细的可变长度令牌表示,先传抽象语义再补细节。
  • 用672个令牌生成10秒视频,比传统方法少8倍,质量相当。
  • 适合需要长视频生成或低计算成本的场景,如文本生成视频。

视觉标记器将高维像素压缩为下游模型可用的表示。现有方法通常将视频视为时空3D网格的标记,要求模型“逐像素”预测所有细节,学习负担重。本文提出VideoFlexTok,采用从粗到细的可变长度标记结构:早期标记捕获语义与运动等抽象信息,后期标记逐步添加细节。生成式解码器支持任意标记数下的真实视频重建。该结构可根据下游需求调整标记数量,在相同预算下处理更长视频。在类别和文本到视频生成任务中,其训练效率显著更高——仅用1.1B参数模型即达到5.2B参数模型的生成质量(gFVD和ViCLIP分数),且可在仅672个令牌下训练10秒81帧视频,比3D网格标记器少8倍。

原文摘要 · Abstract (English)

Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner -- where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.

视频生成令牌化扩散模型长视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。