通过消息协同压缩视觉令牌,提升多模态模型推理效率与准确性。
Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression

- 以注意力消息为单位构建核心子集,避免重复覆盖关键区域。
- 在不训练的情况下实现97.78%性能保留,仅用20%的视觉令牌。
- 适合需要高效部署的大规模多模态模型开发者使用。
现代视觉语言模型将高分辨率图像转为长序列视觉令牌,每个令牌均需经过语言解码器并驻留于提示键值缓存中,显著增加推理开销,促使采用激进的视觉压缩策略。现有基于评分的方法独立评估每个令牌的重要性并保留前K个。然而,文本查询依赖的是视觉群体的集体、有符号注意力消息,而非孤立的图像块。因此,同等大小的前K集合可能重复覆盖某一显著区域,遗漏稀疏但互补的信息,并丢弃被移除群体所携带的消息。为此,本文将忠实的视觉压缩建模为解码器消息的核心子集构造问题,提出无需训练的接地消息核心子集剪枝(GMC)方法:联合分配查询相关、外观及坐标感知证据的支持,并将被移除状态传输至选定代表在原始多模态位置后进行物理压缩,恢复原生注意力机制。该方法将忠实压缩分解为两个耦合组件:选择能覆盖所需消息模式的载体,以及在这些载体上实现有符号群体消息。进一步推导了误差与有符号消息失真、视觉创新性及候选边距稳定性之间的理论边界。跨多个视觉语言模型家族和多样基准的实验表明,GMC-H2在使用80.2%更少视觉令牌的情况下,仍保持Qwen2.5-VL-7B模型97.78%的全相对平均性能;而GMC-L16达到100.36%。受控干预验证了集体支持与群体实现共同驱动性能提升。
原文摘要 · Abstract (English)
Modern vision language models (VLMs) turn high-resolution images into long sequences of visual tokens. Every token traverses the language decoder and persists in its prompt KV cache, inflating inference cost and motivating aggressive visual compression. Existing score-based methods assign each token an independent importance score and retain the Top-K. However, text queries consume collective, signed attention messages from the visual population, not isolated patches. Consequently, equally sized Top-K sets can repeatedly cover one salient region, omit sparse but complementary evidence and discard information carried by the removed population. We therefore formulate faithful visual compression as constructing a compact coreset for decoder messages, and introduce our training-free Grounded Message Coreset Pruning (GMC) which jointly allocates support across query-grounded, appearance, and coordinate-aware evidence, then transports discarded states into selected representatives at their original multimodal positions before physical compaction and native attention resume. This decomposes faithful compression into two coupled components, including selecting carriers that cover the required message modes and realizing the signed population message on those carriers. We further derive bounds connecting their errors to signed-message distortion, visual innovation, and candidate-margin stability. Experiments across multiple VLM families and diverse benchmarks demonstrate strong performance, with GMC-H2 retaining 97.78% Full-relative mean capability on Qwen2.5-VL-7B using 80.2% fewer visual tokens, while GMC-L16 reaches 100.36%. Controlled interventions verify that collective support and population realization jointly drive these gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。