用离散码本+自回归先验，实现0.06比特以下视频压缩。
Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression
- 采用向量量化码本与自回归先验，避免传统率失真优化难题。
- 在64×64分辨率下达到0.043-0.064比特/像素，比H.265低5-7.6倍。
- 码本使用呈幂律分布，熵效率达70%-85%，适合超低码率场景。
基于连续潜变量的可学习视频编码器在低于0.1比特/像素时难以稳定运行：缺乏可微分的率信号，导致拉格朗日优化无法在极端压缩比下有效权衡重建质量与码率。本文证明，离散潜变量可彻底规避此限制。在向量量化（VQ）编码器中，码本大小K带来每符号最多log₂K比特的信息上限；而学习到的自回归先验利用码字使用频率的非均匀性——我们发现其符合幂律分布——使实际码率远低于该上限，且无需率惩罚调参。基于MS-VQ-VAE架构，在统一训练协议下对K∈{128, 256, 512, 1024}进行测试，获得四条率失真（RD）曲线上的操作点。我们识别并解决了关键训练不稳定性：基于梯度的VQ在K≤512时会灾难性崩溃，而采用EMA稳定码本更新并结合死码重启机制，可保持所有配置下的完全码本利用率。在500个UCF101测试片段（64×64，32帧）上，模型实现0.043–0.064比特/像素的压缩率，比H.264实际下限低3.3–5倍，比H.265下限低5–7.6倍。所有MS-VQ-VAE配置在感知质量（LPIPS）上均优于H.265 CRF 36，尽管仅使用其5–7.6倍的比特数。当K=1024时，模型在LPIPS上超越H.265 CRF 36达0.072绝对值，同时比特数仅为后者的5.1倍。码本分析确认幂律指数分布及70%–85%的熵效率，验证该流程为合理可学习的熵编码方案。
原文摘要 · Abstract (English)
Learned video codecs based on continuous latent representations struggle to operate reliably below 0.1 bits per pixel~(bpp): without a differentiable rate signal, Lagrangian optimisation cannot effectively trade reconstruction quality for bitrate at extreme compression ratios. We demonstrate that discrete latent representations sidestep this limitation entirely. In a vector-quantized~(VQ) codec, the codebook size~$K$ imposes a hard information ceiling of $\log_2 K$ bits per symbol; a learned autoregressive prior then exploits the non-uniform distribution of code usage -- which we show follows a power law -- to push actual bitrates well below this ceiling, without any rate-penalty tuning. Building on the MS-VQ-VAE architecture introduced in~\cite{kotthapalli2026msvqvae}, we sweep $K \in \{128, 256, 512, 1024\}$ under a uniform training protocol to trace four operating points on the rate-distortion~(RD) curve. We identify and resolve a critical training instability: gradient-based VQ collapses catastrophically at $K \leq 512$, whereas EMA-stabilised codebook updates with dead-code restart maintain full utilisation across all configurations. On 500 UCF101 test clips ($64\!\times\!64$, 32~frames), our models operate at 0.043-0.064~bpp -- 3.3-5$\times$ below H.264's practical floor and $5$-$7.6\times$ below H.265's floor at this resolution. Every MS-VQ-VAE configuration outperforms H.265 CRF\,36 on perceptual quality (LPIPS) despite using $5$-$7.6\times$ fewer bits. At $K{=}1024$, the model surpasses H.265 CRF\,36 on LPIPS by a margin of 0.072 absolute while using $5.1\times$ fewer bits. Codebook analysis confirms power-law index distributions and 70-85\% entropy efficiency, establishing the pipeline as a principled learned entropy coder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。