为多模态大模型量身定制图像压缩方案,节省带宽同时保持模型性能。
When MLLMs Meet Compression Distortion: A Coding Paradigm Tailored to MLLMs
- 基于CLIP浅层注意力生成重要性图,动态分配比特保护关键特征。
- 采用多层级损失函数,在低层细节与高层语义间实现均衡重建。
- 在保持模型任务性能前提下,最高可减少35.99%码率,适合边缘部署。
随着强大多模态大语言模型(MLLMs)在云端部署日益增多,高效传输边缘设备的图像、视频等信号输入亟需有效压缩技术以降低带宽消耗。然而,传统图像编码器针对人类视觉系统优化,注重保真度,不适用于需综合考虑多种下游任务的MLLMs。本文首次系统分析了压缩失真对主流MLLMs的影响,发现不同层级图像特征受压缩影响程度不均,导致下游任务表现差异。为此,提出专为MLLMs设计的图像编码器CoTAM,通过CLIP浅层注意力生成重要性图指导比特分配,保护关键语义区域;解码端集成轻量适配器与多层级损失函数,确保低层细节和高层语义的准确重建,提升跨层级特征合成能力。大量实验表明,该方法在维持原有任务性能基础上,最高可节省35.99%码率,优于现有最优神经编码器。
原文摘要 · Abstract (English)
The increasing deployment of powerful Multimodal Large Language Models (MLLMs), typically hosted on cloud platforms, urgently requires effective compression techniques to efficiently transmit signal inputs (e.g., images, videos) from edge devices with minimal bandwidth usage. However, conventional image codecs are optimized for fidelity to serve the Human Visual System (HVS) and ill-suited for MLLMs, in which diverse downstream tasks are jointly considered. In this paper, we first systematically analyze the impact of compression artifacts on several mainstream MLLMs. We find that: Compression distortion unevenly impacts different-level image features, leading to varying effects on MLLMs' downstream tasks depending on their feature-level reliance. Motivated by this discovery, we propose an image Codec TAilored to MLLMs (CoTAM) designed to adaptively protect multi-level features and suit different demands of downstream tasks. The encoder leverages CLIP's shallow-layer attention to generate an importance map for bit allocation, preserving critical semantic regions. Concurrently, the decoder integrates a lightweight adapter with a multi-level loss function to ensure the faithful reconstruction both of low-level details and high-level semantic context for robust synthesis of cross-level features. Extensive experiments validate that our method achieves up to 35.99\% bitrate saving while maintaining the same performance on the MLLM tasks, outperforming previous SOTA neural codecs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。