用生成模型的潜在空间压缩图像,实现超低码率下高保真与高真实感
Generative Latent Coding for Ultra-Low Bitrate Image Compression
- 在生成式VQ-VAE的潜在空间进行变换编码,更契合人眼感知
- 自然图像压缩低于0.04 bpp仍保持高视觉质量,人脸图像低于0.01 bpp
- 适合对压缩率和画质兼得有需求的研究者与开发者
现有图像压缩方法多在像素空间进行变换编码以降低空间冗余,但在低码率下难以同时实现高真实感与高保真,因像素空间失真与人眼感知不一致。为此,本文提出生成式潜在编码(GLC)架构,将变换编码从像素空间转移到生成式向量量化变分自编码器(VQ-VAE)的潜在空间。该潜在空间具有更强稀疏性、更丰富的语义信息且更符合人类感知,有利于实现高真实感与高保真压缩。此外,引入类别化超模块降低超信息比特开销,并采用基于代码预测的监督机制增强语义一致性。实验表明,本方法在自然图像上压缩码率低于0.04 bpp,在人脸图像上低于0.01 bpp仍保持高视觉质量;在CLIC2020测试集上,达到与MS-ILLM相当的FID,但比特数减少45%。强大的生成潜在空间支持多种应用,如图像修复与风格迁移。代码已开源:https://github.com/jzyustc/GLC。
原文摘要 · Abstract (English)
Most existing image compression approaches perform transform coding in the pixel space to reduce its spatial redundancy. However, they encounter difficulties in achieving both high-realism and high-fidelity at low bitrate, as the pixel-space distortion may not align with human perception. To address this issue, we introduce a Generative Latent Coding (GLC) architecture, which performs transform coding in the latent space of a generative vector-quantized variational auto-encoder (VQ-VAE), instead of in the pixel space. The generative latent space is characterized by greater sparsity, richer semantic and better alignment with human perception, rendering it advantageous for achieving high-realism and high-fidelity compression. Additionally, we introduce a categorical hyper module to reduce the bit cost of hyper-information, and a code-prediction-based supervision to enhance the semantic consistency. Experiments demonstrate that our GLC maintains high visual quality with less than 0.04 bpp on natural images and less than 0.01 bpp on facial images. On the CLIC2020 test set, we achieve the same FID as MS-ILLM with 45% fewer bits. Furthermore, the powerful generative latent space enables various applications built on our GLC pipeline, such as image restoration and style transfer. The code is available at https://github.com/jzyustc/GLC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。