首个实现实时扩散模型视频压缩的框架,显著降低延迟与失真。
DiffVC-RT: Towards Practical Real-Time Diffusion-based Perceptual Neural Video Compression
- 通过模块替换与剪枝优化架构,减少计算量并保留结构信息。
- 在HEVC数据集上实现80.1%码率节省,720p下编码解码速度达206/30fps。
- 适合需要低延迟、高感知质量的实时视频传输场景。
基于扩散模型的神经视频压缩(NVC)在实际部署中面临严重信息损失、推理延迟过高及时间不一致等问题。为此,我们提出DiffVC-RT,首个实现实时扩散模型感知视频压缩的框架。首先,设计高效且信息丰富的模型架构,通过关键模块替换与剪枝,显著降低计算复杂度并缓解结构信息丢失。其次,为解决生成闪烁伪影,提出显式与隐式一致性建模:在U-Net中引入零成本在线时间位移模块,并结合混合隐式一致性约束以增强时序一致性。最后,构建异步并行解码流水线,采用混合半精度与批维度时间位移设计,实现异步潜在表示解码与并行帧重建。实验表明,DiffVC-RT在HEVC数据集上相比VTM-17.0实现80.1%的LPIPS码率节省,720p视频在NVIDIA H800 GPU上达到206/30fps的实时编码与解码速度,标志着扩散模型视频压缩的重要突破。
原文摘要 · Abstract (English)
The practical deployment of diffusion-based Neural Video Compression (NVC) faces critical challenges, including severe information loss, prohibitive inference latency, and poor temporal consistency. To bridge this gap, we propose DiffVC-RT, the first framework designed to achieve real-time diffusion-based perceptual NVC. First, we introduce an Efficient and Informative Model Architecture. Through strategic module replacements and pruning, this architecture significantly reduces computational complexity while mitigating structural information loss. Second, to address generative flickering artifacts, we propose Explicit and Implicit Consistency Modeling. We enhance temporal consistency by explicitly incorporating a zero-cost Online Temporal Shift Module within the U-Net, complemented by hybrid implicit consistency constraints. Finally, we present an Asynchronous and Parallel Decoding Pipeline incorporating Mixed Half Precision, which enables asynchronous latent decoding and parallel frame reconstruction via a Batch-dimension Temporal Shift design. Experiments show that DiffVC-RT achieves 80.1% bitrate savings in terms of LPIPS over VTM-17.0 on HEVC dataset with real-time encoding and decoding speeds of 206 / 30 fps for 720p videos on an NVIDIA H800 GPU, marking a significant milestone in diffusion-based video compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。