用视觉提示让大模型预测图像残差,实现更优无损压缩。
Large Language Model for Lossless Image Compression with Visual Prompts
- 用损失性重建生成视觉提示,引导大模型理解图像上下文。
- 在多个基准数据集上超越传统与学习型编码器的压缩效果。
- 适用于医疗、屏幕内容等多领域图像,拓展性强。
深度学习的进展推动了无损图像压缩的显著进步。随着大语言模型(LLMs)的出现,初步尝试利用其丰富的先验知识来提升无损压缩性能,尤其在熵模型方面。然而,如何弥合LLMs中的文本先验知识与无损图像压缩之间的鸿沟仍是一大挑战。为此,本文提出一种新范式:将大语言模型与视觉提示结合。具体而言,首先生成输入图像的损失性重建作为视觉提示,并从中提取特征作为视觉嵌入供大模型使用;随后将原始图像与损失性重建的残差连同视觉嵌入一同输入大模型,使其作为熵模型预测残差的概率分布。在多个基准数据集上的大量实验表明,该方法实现了当前最优的压缩性能,超越了传统及基于学习的无损图像编解码器。此外,该方法可轻松扩展至医学图像和屏幕内容等其他领域,表现优异。这些结果凸显了大语言模型在无损图像压缩中的潜力,或可激发相关领域的进一步研究。
原文摘要 · Abstract (English)
Recent advancements in deep learning have driven significant progress in lossless image compression. With the emergence of Large Language Models (LLMs), preliminary attempts have been made to leverage the extensive prior knowledge embedded in these pretrained models to enhance lossless image compression, particularly by improving the entropy model. However, a significant challenge remains in bridging the gap between the textual prior knowledge within LLMs and lossless image compression. To tackle this challenge and unlock the potential of LLMs, this paper introduces a novel paradigm for lossless image compression that incorporates LLMs with visual prompts. Specifically, we first generate a lossy reconstruction of the input image as visual prompts, from which we extract features to serve as visual embeddings for the LLM. The residual between the original image and the lossy reconstruction is then fed into the LLM along with these visual embeddings, enabling the LLM to function as an entropy model to predict the probability distribution of the residual. Extensive experiments on multiple benchmark datasets demonstrate our method achieves state-of-the-art compression performance, surpassing both traditional and learning-based lossless image codecs. Furthermore, our approach can be easily extended to images from other domains, such as medical and screen content images, achieving impressive performance. These results highlight the potential of LLMs for lossless image compression and may inspire further research in related directions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。