将视频的外观与运动分开编码，提升生成效率与效果。
DeRA: Decoupled Representation Alignment for Video Tokenization
- 分离视频的外观与运动流，分步对齐预训练模型。
- 在UCF-101上比SOTA低25%的rFVD，性能显著提升。
- 适合视频生成、动作识别等需要高效表征的任务。
本文提出DeRA，一种新型1D视频分词器，通过解耦视频编码中的时空表征学习，实现更优的训练效率与性能。DeRA维持紧凑的1D潜在空间，将视频编码分解为外观流与运动流，分别与预训练视觉基础模型对齐，以独立捕捉视频的空间语义与时间动态。针对异构监督带来的梯度冲突，进一步提出对称对齐-冲突投影（SACP）模块，主动通过抑制冲突方向的梯度分量来重构梯度。大量实验表明，DeRA在UCF-101上的rFVD指标比此前最优方法LARP降低25%。此外，使用DeRA进行自回归视频生成，在UCF-101类条件生成与K600帧预测任务上均达到新SOTA水平。
原文摘要 · Abstract (English)
This paper presents DeRA, a novel 1D video tokenizer that decouples the spatial-temporal representation learning in video tokenization to achieve better training efficiency and performance. Specifically, DeRA maintains a compact 1D latent space while factorizing video encoding into appearance and motion streams, which are aligned with pretrained vision foundation models to capture the spatial semantics and temporal dynamics in videos separately. To address the gradient conflicts introduced by the heterogeneous supervision, we further propose the Symmetric Alignment-Conflict Projection (SACP) module that proactively reformulates gradients by suppressing the components along conflicting directions. Extensive experiments demonstrate that DeRA outperforms LARP, the previous state-of-the-art video tokenizer by 25% on UCF-101 in terms of rFVD. Moreover, using DeRA for autoregressive video generation, we also achieve new state-of-the-art results on both UCF-101 class-conditional generation and K600 frame prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。