统一多模态图像压缩,提升各类编码器的画质与效率。
UniMIC: Towards Universal Multi-modality Perceptual Image Compression
- 构建视觉编码库,复用多种现成编码器
- 通过文本提示实现多粒度编码,增强感知重建
- 利用扩散模型修复画质,适配各类压缩场景
我们提出UniMIC,一个统一的多模态图像压缩框架,旨在通过挖掘跨模态生成先验,同时优化多种图像编码器的率失真感知(RDP)性能。不同于需从头设计编码器的方法,UniMIC引入视觉编码库,直接复用大量代表性编码器作为基础编码器,适用于各类实际应用。此外,提出多粒度文本编码机制,设计可变长度内容提示和压缩提示,借助多模态条件生成辅助感知重建。特别地,提出通用感知补偿器,在解码端利用Stable Diffusion中的文本辅助扩散先验,统一提升所有基础编码器解码图像的感知质量。三种策略协同作用,显著改善了传统与可学习编码器在不同压缩成本(如超低比特率)下的RDP性能。代码将开源于https://github.com/Amygyx/UniMIC。
原文摘要 · Abstract (English)
We present UniMIC, a universal multi-modality image compression framework, intending to unify the rate-distortion-perception (RDP) optimization for multiple image codecs simultaneously through excavating cross-modality generative priors. Unlike most existing works that need to design and optimize image codecs from scratch, our UniMIC introduces the visual codec repository, which incorporates amounts of representative image codecs and directly uses them as the basic codecs for various practical applications. Moreover, we propose multi-grained textual coding, where variable-length content prompt and compression prompt are designed and encoded to assist the perceptual reconstruction through the multi-modality conditional generation. In particular, a universal perception compensator is proposed to improve the perception quality of decoded images from all basic codecs at the decoder side by reusing text-assisted diffusion priors from stable diffusion. With the cooperation of the above three strategies, our UniMIC achieves a significant improvement of RDP optimization for different compression codecs, e.g., traditional and learnable codecs, and different compression costs, e.g., ultra-low bitrates. The code will be available in https://github.com/Amygyx/UniMIC .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。