用Stable Diffusion实现AI图像分层压缩,低码率下保真度高。
Stable Diffusion is a Natural Cross-Modal Decoder for Layered AI-generated Image Compression
- 将AI图像分为语义、结构、纹理三层编码,分别对应文本提示、边缘图、颜色映射。
- 在低于0.02 bpp的极低码率下,仍能还原语义与视觉细节。
- 支持无需完整解码的下游编辑,适合高效传输与交互式应用。
人工智能生成内容(AIGC)的快速发展带来了大量AI生成图像(AIGIs)的传输与压缩需求,但相关压缩研究仍显不足。为此,我们提出一种可扩展的跨模态压缩框架,融合多个人类可理解模态,高效捕捉并传递AIGIs的关键视觉信息。该框架将图像编码为三层比特流:语义层通过文本提示传递高层语义;结构层利用边缘或骨架图捕获空间细节;纹理层通过颜色映射保留局部纹理。以Stable Diffusion为后端,当这些先验信息被编码时,其可自然充当解码器。定性和定量结果表明,本方法在极低码率(<0.02 bpp)下仍能有效恢复语义与视觉细节,性能优于基线方法。此外,该框架支持无需完整解码的下游编辑,为未来AIGI压缩研究开辟新方向。
原文摘要 · Abstract (English)
Recent advances in Artificial Intelligence Generated Content (AIGC) have garnered significant interest, accompanied by an increasing need to transmit and compress the vast number of AI-generated images (AIGIs). However, there is a noticeable deficiency in research focused on compression methods for AIGIs. To address this critical gap, we introduce a scalable cross-modal compression framework that incorporates multiple human-comprehensible modalities, designed to efficiently capture and relay essential visual information for AIGIs. In particular, our framework encodes images into a layered bitstream consisting of a semantic layer that delivers high-level semantic information through text prompts; a structural layer that captures spatial details using edge or skeleton maps; and a texture layer that preserves local textures via a colormap. Utilizing Stable Diffusion as the backend, the framework effectively leverages these multimodal priors for image generation, effectively functioning as a decoder when these priors are encoded. Qualitative and quantitative results show that our method proficiently restores both semantic and visual details, competing against baseline approaches at extremely low bitrates ( <0.02 bpp). Additionally, our framework facilitates downstream editing applications without requiring full decoding, thereby paving a new direction for future research in AIGI compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。