用提示词指导压缩，只保留关键图像区域，提升效率。
Prompt-Guided Prefiltering for VLM Image Compression
- 根据文本提示筛选图像重点区域，其余区域平滑处理
- 在多个VQA任务上平均码率降低25%-50%，准确率不变
- 可适配各类编码器，适合机器视觉压缩场景
大型视觉语言模型（VLM）的快速发展推动了图像理解与视觉问答（VQA）等应用。查询图像通常上传至云端，由部署在云上的VLM处理，因此高效图像压缩至关重要。然而，传统以人为中心的编码器在此场景下表现不佳，因其保留了大量任务无关细节。现有面向机器的图像编码（ICM）方法也存在局限，因假设下游任务固定，无法适应开放式的提示驱动型VLM任务。本文提出一种轻量、即插即用的提示引导预过滤模块，能识别与文本提示最相关的图像区域，从而提升压缩效率。该模块在保留关键信息的同时，对非相关区域进行平滑处理，且与编码器无关，可应用于传统及学习型编码器之前。在多个VQA基准测试中，本方法实现平均25%-50%的码率降低，同时保持原有任务精度。源代码已公开于https://github.com/bardia-az/pgp-vlm-compression。
原文摘要 · Abstract (English)
The rapid progress of large Vision-Language Models (VLMs) has enabled a wide range of applications, such as image understanding and Visual Question Answering (VQA). Query images are often uploaded to the cloud, where VLMs are typically hosted, hence efficient image compression becomes crucial. However, traditional human-centric codecs are suboptimal in this setting because they preserve many task-irrelevant details. Existing Image Coding for Machines (ICM) methods also fall short, as they assume a fixed set of downstream tasks and cannot adapt to prompt-driven VLMs with an open-ended variety of objectives. We propose a lightweight, plug-and-play, prompt-guided prefiltering module to identify image regions most relevant to the text prompt, and consequently to the downstream task. The module preserves important details while smoothing out less relevant areas to improve compression efficiency. It is codec-agnostic and can be applied before conventional and learned encoders. Experiments on several VQA benchmarks show that our approach achieves a 25-50% average bitrate reduction while maintaining the same task accuracy. Our source code is available at https://github.com/bardia-az/pgp-vlm-compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。