首个专为视频压缩设计的扩散模型,实现超低码率下的高质量重建。
Generative Video Compression with Adaptive Score Distillation

- 从零训练压缩专用扩散模型,像素空间直接重建
- 自适应蒸馏技术提升推理速度,保持运动连贯性
- 仅4780万参数,单步解码1080p视频达15.1帧/秒
扩散模型在超低码率下展现出强大的视频压缩生成能力。现有基于扩散的视频编码器沿用原本为文本条件生成设计的基础模型,而专为压缩任务设计并训练的扩散模型仍待探索。为此,我们提出生成式视频编码器GenVC,基于从零开始训练的、面向压缩的视频扩散模型。据我们所知,这是首个面向压缩的视频扩散模型。该模型直接在像素空间中实现,采用全局到局部的层次结构以恢复精细的时空细节,从而实现从压缩表示中高质量的生成重建。为加速推理,我们使用分布匹配蒸馏(DMD)将多步模型压缩为单步。然而,直接应用DMD会导致学生模型产生运动停滞的重建结果。我们发现这是教师端引导失效所致:一旦学生引入的扰动超出冻结教师的训练区域,其引导可能变得误导,使DMD更新强化而非修正学生漂移。为打破这一反馈循环,我们提出自适应得分蒸馏(Adaptive Score Distillation),根据更新与真实方向的一致性进行门控,从而在保持运动连贯性的前提下实现高质量重建。实验表明,GenVC在超低码率下达到当前最优感知质量,在相同LPIPS下平均码率降低62.5%,在相同FID下降低71.3%(相比GLVC)。不同于依赖数十亿参数预训练主干的以往编码器,我们的扩散模型仅含478.0M参数,并可在A100 GPU上以15.1 fps单步解码1080p视频。
原文摘要 · Abstract (English)
Diffusion models provide strong generative capabilities for video compression at ultra-low bitrates. Existing diffusion-based video codecs adapt base models originally developed for text-conditioned generation, whereas diffusion models designed and trained specifically for compression remain unexplored. To fill this gap, we introduce our Generative Video Codec (GenVC), built on a video diffusion model trained from scratch for compression. To our knowledge, this is the first compression-oriented video diffusion model. We realize this model directly in pixel space with a global-to-local hierarchy that recovers fine spatio-temporal details, enabling high-quality generative reconstruction from compressed representations. To accelerate inference, we distill the multi-step model into one step using distribution matching distillation (DMD). Applying DMD directly, however, drives the student toward motion-stalled reconstructions. We trace this to a teacher-side guidance failure: once student-induced perturbations leave the frozen teacher's training region, its guidance can become misleading, causing DMD updates to reinforce rather than correct the student drift. To break the resulting feedback loop, we propose Adaptive Score Distillation, which gates DMD updates according to their alignment with the ground-truth direction, enabling high-quality reconstruction with coherent motion. Experimental results show that GenVC achieves state-of-the-art perceptual quality at ultra-low bitrates, with average bitrate savings of 62.5% at matched LPIPS and 71.3% at matched FID over GLVC. Unlike prior codecs that inherit billion-scale pretrained backbones, our diffusion model has only 478.0M parameters and decodes 1080p video in a single step at 15.1 fps on an A100 GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。