一张压缩图同时满足人眼观看和机器分析,无需重新编码。
Unified Coding for Both Human Perception and Generalized Machine Analytics with CLIP Supervision
- 用CLIP模型提供跨任务监督,提升压缩的通用性。
- 单个码流可支持未见过的机器视觉任务,且人眼感知质量好。
- 自监督训练,不依赖具体下游任务,适合多场景应用。
图像压缩模型长期面临适应性与泛化能力不足的问题,传统解码码流通常仅服务于人眼或特定机器任务,难以保留未知视觉任务所需信息。本文提出统一且通用的图像编码方法UG-ICM,通过引入对比语言-图像预训练(CLIP)模型作为训练约束,实现对人类视觉感知与机器视觉任务的双重支持。采用从全局到实例级别的CLIP监督,增强模型对不同粒度语义的理解,提高泛化能力。为实现单一码流同时适配人眼与机器需求,设计条件解码策略,根据人类或机器偏好生成对应版本。整个UG-ICM模型在自监督模式下训练,无需了解任何具体下游任务。大量实验表明,该方法在多种未见过的机器分析任务中表现优异,同时保持高质量的人眼感知效果。
原文摘要 · Abstract (English)
The image compression model has long struggled with adaptability and generalization, as the decoded bitstream typically serves only human or machine needs and fails to preserve information for unseen visual tasks. Therefore, this paper innovatively introduces supervision obtained from multimodal pre-training models and incorporates adaptive multi-objective optimization tailored to support both human visual perception and machine vision simultaneously with a single bitstream, denoted as Unified and Generalized Image Coding for Machine (UG-ICM). Specifically, to get rid of the reliance between compression models with downstream task supervision, we introduce Contrastive Language-Image Pre-training (CLIP) models into the training constraint for improved generalization. Global-to-instance-wise CLIP supervision is applied to help obtain hierarchical semantics that make models more generalizable for the tasks relying on the information of different granularity. Furthermore, for supporting both human and machine visions with only a unifying bitstream, we incorporate a conditional decoding strategy that takes as conditions human or machine preferences, enabling the bitstream to be decoded into different versions for corresponding preferences. As such, our proposed UG-ICM is fully trained in a self-supervised manner, i.e., without awareness of any specific downstream models and tasks. The extensive experiments have shown that the proposed UG-ICM is capable of achieving remarkable improvements in various unseen machine analytics tasks, while simultaneously providing perceptually satisfying images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。