让视觉语言模型用极少图像令牌完成高效推理
ETC: Extreme Token Compression via Task-aware Visual Information Distillation in VLMs
- 通过任务感知信息蒸馏，压缩图像令牌数
- 单令牌压缩仍保持强任务性能，降低缓存开销
- 适合需要高效推理的多模态应用
在视觉语言模型中，高分辨率图像产生大量视觉令牌，导致推理时计算成本高、键值缓存开销大。为此，我们提出极端令牌压缩（ETC）框架，基于变分信息蒸馏原则，在减少输入令牌数量的同时最小化任务损失。从信息论视角看，最小化任务损失要求紧凑表示保留与任务相关视觉信息的指令感知充分统计量。实际中，ETC利用文本到图像交叉注意力对原始视觉特征加权，近似潜在的指令感知预测统计量；同时引入变分信息蒸馏机制，使紧凑表示能保留恢复该统计量的关键信息。在LLaVA-1.5-7B和Qwen3-VL-2B上的实验表明，即使在单令牌压缩下，ETC仍有效，显著降低键值缓存开销且保持强大任务性能。
原文摘要 · Abstract (English)
In Vision-Language Models (VLMs), high-resolution images produce a large number of visual tokens, resulting in high computational costs and KV-cache overhead during inference. To address this problem, we propose an Extreme Token Compression (ETC) framework that minimizes task loss when reducing the number of input tokens based on the principle of variational information distillation. Specifically, from an information-theoretic perspective, we show that minimizing task loss requires the compact representation to preserve the instruction-aware sufficient statistic of the task-relevant visual information for prediction. In practice, ETC leverages text-to-image cross-attention to weight the original visual features to approximate the latent instruction-aware predictive statistic. Moreover, ETC introduces a variational information distillation, enabling the compact representation to preserve the essential information to recover this predictive statistic. Experiments on LLaVA-1.5-7B and Qwen3-VL-2B show that ETC remains effective even under single-token compression, substantially reducing KV-cache overhead while retaining strong task performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。