将视频分解为结构与动态两个独立潜在空间，提升生成质量与效率。
VidTwin: Video VAE with Decoupled Structure and Dynamics
- 分离视频的全局结构与快速动态，分别建模
- 在MCL-JCV数据集上压缩率0.20%，PSNR达28.14
- 适合需要高效、可解释视频生成的研究者
近期视频自编码器（Video AEs）在视频生成的质量与效率上取得显著进展。本文提出一种新型紧凑型视频自编码器VidTwin，将视频分解为两个独立的潜在空间：结构潜向量捕捉整体内容与全局运动，动态潜向量表征细粒度细节与快速运动。具体而言，模型采用编码器-解码器主干网络，并引入两个子模块分别提取上述潜空间：第一个子模块使用Q-Former提取低频运动趋势，再通过下采样块去除冗余细节；第二个子模块沿空间维度平均潜向量以捕捉快速运动。大量实验表明，VidTwin在保持高重建质量的同时实现0.20%的高压缩率（在MCL-JCV数据集上PSNR达28.14），并在下游生成任务中表现出高效性与有效性。此外，模型具备可解释性与可扩展性，为未来视频潜在表示与生成研究提供新方向。
原文摘要 · Abstract (English)
Recent advancements in video autoencoders (Video AEs) have significantly improved the quality and efficiency of video generation. In this paper, we propose a novel and compact video autoencoder, VidTwin, that decouples video into two distinct latent spaces: Structure latent vectors, which capture overall content and global movement, and Dynamics latent vectors, which represent fine-grained details and rapid movements. Specifically, our approach leverages an Encoder-Decoder backbone, augmented with two submodules for extracting these latent spaces, respectively. The first submodule employs a Q-Former to extract low-frequency motion trends, followed by downsampling blocks to remove redundant content details. The second averages the latent vectors along the spatial dimension to capture rapid motion. Extensive experiments show that VidTwin achieves a high compression rate of 0.20% with high reconstruction quality (PSNR of 28.14 on the MCL-JCV dataset), and performs efficiently and effectively in downstream generative tasks. Moreover, our model demonstrates explainability and scalability, paving the way for future research in video latent representation and generation. Check our project page for more details: https://vidtwin.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。