用CLIP字典实现多图语义压缩，提升压缩率且不损失语义质量。
SMIC: Semantic Multi-Item Compression based on CLIP dictionary
- 基于CLIP隐空间设计语义字典，支持图像间语义加减操作。
- 压缩率比当前最优生成式方法提升约10^-5 BPP/图。
- 字典具语义性，可作图像内容的语义投影工具，适合图像集合压缩场景。
语义压缩通过以语义保真度替代传统均方误差作为失真度量，日益受到关注。现有方法多依赖基础模型CLIP。本文将该方案拓展至图像集合压缩，编码阶段考虑图像间的冗余信息。首先证明了CLIP隐空间支持便捷的语义加减操作；基于此构建基于字典的多图编码器，在压缩率上优于当前最优生成式编码器，每张图像提升约10^-5 BPP，同时保持语义保真度。此外，所学习的字典具有语义特性，可作为图像语义内容的投影器。
原文摘要 · Abstract (English)
Semantic compression, a compression scheme where the distortion metric, typically MSE, is replaced with semantic fidelity metrics, tends to become more and more popular. Most recent semantic compression schemes rely on the foundation model CLIP. In this work, we extend such a scheme to image collection compression, where inter-item redundancy is taken into account during the coding phase. For that purpose, we first show that CLIP's latent space allows for easy semantic additions and subtractions. From this property, we define a dictionary-based multi-item codec that outperforms state-of-the-art generative codec in terms of compression rate, around $10^{-5}$ BPP per image, while not sacrificing semantic fidelity. We also show that the learned dictionary is of a semantic nature and works as a semantic projector for the semantic content of images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。