arXiv:2602.04202cs.CV2026-02被引 1

提出统一视频分词框架,分离时空特征提升生成与理解效果。

VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latents

  • 用关键帧+残差令牌分离视频时空表示,降低冗余。
  • 在多个基准上表现更优,如TV-Align准确率高3.4%。
  • 适合视频生成与理解任务,尤其擅长保持运动连贯性。

本文提出VTok,一种统一的视频分词框架,适用于视频生成与理解任务。不同于主流视觉语言系统采用简单帧采样策略,我们通过保留单个关键帧的空间特征,并将后续每帧编码为单一残差令牌,实现时空解耦,获得紧凑而富有表现力的视频分词。实验表明,VTok将视频表征复杂度从帧数与每帧令牌数的乘积降至二者之和,残差令牌能充分捕捉相对于关键帧的视角与运动变化。大量评估显示,相比使用朴素分词的基线,VTok在多种视频理解与文生视频基准上性能显著提升,且每视频令牌序列更短(如在TV-Align基准上准确率提高3.4%,VBench得分提升1.9%)。值得注意的是,由于更一致的时序编码,VTok在文生视频生成中产生更连贯的运动和更强的文本引导效果。我们希望VTok能成为未来视频理解与生成研究的标准分词范式。

原文摘要 · Abstract (English)

This work presents VTok, a unified video tokenization framework that can be used for both generation and understanding tasks. Unlike the leading vision-language systems that tokenize videos through a naive frame-sampling strategy, we propose to decouple the spatial and temporal representations of videos by retaining the spatial features of a single key frame while encoding each subsequent frame into a single residual token, achieving compact yet expressive video tokenization. Our experiments suggest that VTok effectively reduces the complexity of video representation from the product of frame count and per-frame token count to their sum, while the residual tokens sufficiently capture viewpoint and motion changes relative to the key frame. Extensive evaluations demonstrate the efficacy and efficiency of VTok: it achieves notably higher performance on a range of video understanding and text-to-video generation benchmarks compared with baselines using naive tokenization, all with shorter token sequences per video (e.g., 3.4% higher accuracy on our TV-Align benchmark and 1.9% higher VBench score). Remarkably, VTok produces more coherent motion and stronger guidance following in text-to-video generation, owing to its more consistent temporal encoding. We hope VTok can serve as a standardized video tokenization paradigm for future research in video understanding and generation.

视频分词生成模型时空建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。