用生成模型在极低码率下实现高感知质量视频压缩
Group-of-Latents: Perceptual Video Compression at Extreme Bitrates via Masked Latent Generative Modeling

- 将隐空间分为帧内和帧间隐变量,用生成模型重建动态
- 在低于0.005 bpp时仍保持丰富细节和稳定时序
- 适合超低码率视频传输与存储场景
现有视频压缩算法多基于变换与量化,但在极低码率下感知质量优化仍未充分解决。本文提出统一生成框架,利用预训练的扩散变换器(DiT)先验,在极端低码率下实现高感知质量。首先在因果分词器的隐空间中引入灵活的隐变量组(GoL)策略,显式划分帧内(I-latents)与帧间(P-latents)隐变量。深度压缩模块(I-DCM)编码关键帧内隐变量,以最小开销保留感知锚点。在此基础上,基于DiT的统一隐变量去噪模块(U-LDM)从噪声中重构帧内纹理并生成帧间隐变量,实现零额外码率代价的时序动态重建。大量实验表明,该方法在极低码率(如<0.005 bpp)下达到当前最优感知保真度,兼具丰富空间细节与强时序一致性。代码将公开。
原文摘要 · Abstract (English)
Most existing video compression algorithms follow a paradigm of transformation and quantization, optimizing the trade-off between distortion and bitrate. However, extremely low-bitrate compression remains an underexplored frontier where perceptual quality optimization under severely constrained coding resources has not been adequately addressed. In this paper, we propose a unified generative framework that leverages pre-trained Diffusion Transformer (DiT) priors to achieve high perceptual quality at extremely low bitrates. We first introduce a flexible Group-of-Latents (GoL) strategy within the latent space of a causal tokenizer, explicitly partitioning the latent stream into intra $I$-latents and inter $P$-latents. The Deep Compression Module (I-DCM) then encodes key $I$-latents to preserve perceptual anchors with minimal overhead. Building upon these anchors, the DiT-based Unified Latent Denoising Module (U-LDM) refines intra-frame textures and synthesizes $P$-latents from noise, reconstructing temporal dynamics at zero additional bitrate cost. Extensive experiments demonstrate that our method uniquely operates in the extreme-low-bitrate regime (e.g., (<0.005) bpp), achieving state-of-the-art perceptual fidelity with rich spatial details and robust temporal consistency. The code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。