arXiv:2412.18158cs.CVeess.IV2024-12

UniCodec用语义解耦与组合生成,一码通用于人眼和机器视觉。

Semantics Disentanglement and Composition for Universal Image Coding with Efficiently LLM Reasoning and Generative Diffusion

  • 编码器通过LLM生成任务专用标签码本,实现语义解耦压缩。
  • 解码器结合扩散模型生成高质量图像,兼顾人眼与机器需求。
  • 换任务只需换码本,无需重新训练,适应速度快。

学习型图像压缩方法虽表现优异,但通常仅针对人眼感知或特定机器视觉任务优化,限制了通用性,且新任务需高昂重训成本。为此,我们提出UniCodec,一种基于编码器语义解耦与解码器组合生成的新范式通用编解码器。编码器利用大语言模型(LLM)预生成的任务专用标签码本,由接地模型使用对应码本执行任务感知的语义解耦,仅压缩最相关图像区域,显著节省比特数,同时实现零重训快速切换任务——换任务只需更换码本。解码器则通过组合生成,将紧凑解耦组件与生成扩散模型的强大先验结合,重建出细节丰富、质量高、兼顾人眼感知与机器视觉精确特征的完整图像。大量实验表明,UniCodec持续优于现有方法,有效弥合了人本与机本压缩间的差距。

原文摘要 · Abstract (English)

Learned image compression methods have shown impressive performance but are often highly specialized for either human perception or specific machine vision tasks. This specialization limits their versatility and requires costly retraining for new applications. To address this, we introduce UniCodec, a universal codec built on a novel paradigm of semantic disentanglement at the encoder and compositional generation at the decoder. This framework is designed to simultaneously serve both human and machine needs, eliminating the need for task-specific retraining. At the encoder, UniCodec leverages pre-generated, task-specific label codebooks created by a Large Language Model (LLM). For any given task, a grounding model uses the corresponding codebook to perform task-aware disentanglement, compressing only the most relevant image regions. This mechanism not only saves significant bits but is also the key to our system's rapid, zero-retraining adaptation: switching to a new task is as simple as selecting a new codebook. The decoder then performs compositional generation: it combines the compact, disentangled components with powerful priors from a generative diffusion model. This process reconstructs a high-quality, complete image optimized with rich detail for human perception and precise features for machine vision tasks. Extensive experiments demonstrate that UniCodec consistently outperforms existing methods, effectively bridging the gap between human-centric and machine-centric compression.

图像压缩语义解耦扩散模型通用编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。