将视频分解为关键帧、运动和残差，实现更紧凑的重建表示。
DeCo-VAE: Learning Compact Latents for Video Reconstruction via Decoupled Representation
- 将视频内容显式分解为关键帧、运动和残差三部分分别建模。
- 在多个数据集上重建质量优于现有VAE方法，且潜空间更紧凑。
- 适合需要高效视频压缩与重建的场景，如流媒体与存储优化。
现有视频变分自编码器（VAE）普遍忽视帧间内容相似性，导致潜变量建模冗余。本文提出解耦变分自编码器（DeCo-VAE），实现紧凑的潜表示。不直接编码RGB像素，而是通过显式解耦将视频内容分解为关键帧、运动和残差三部分，并为每部分学习专用潜变量。为避免跨组件干扰，设计专用编码器分别处理各成分，并采用共享3D解码器以保持重建过程中的时空一致性。进一步提出解耦训练策略：训练时冻结部分编码器，逐个更新其余编码器，确保训练稳定并准确学习静态与动态特征。大量定量与定性实验表明，DeCo-VAE在视频重建性能上显著优于现有方法。
原文摘要 · Abstract (English)
Existing video Variational Autoencoders (VAEs) generally overlook the similarity between frame contents, leading to redundant latent modeling. In this paper, we propose decoupled VAE (DeCo-VAE) to achieve compact latent representation. Instead of encoding RGB pixels directly, we decompose video content into distinct components via explicit decoupling: keyframe, motion and residual, and learn dedicated latent representation for each. To avoid cross-component interference, we design dedicated encoders for each decoupled component and adopt a shared 3D decoder to maintain spatiotemporal consistency during reconstruction. We further utilize a decoupled adaptation strategy that freezes partial encoders while training the others sequentially, ensuring stable training and accurate learning of both static and dynamic features. Extensive quantitative and qualitative experiments demonstrate that DeCo-VAE achieves superior video reconstruction performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。