arXiv:2512.08240cs.CVcs.AI2025-12中稿 · CVPR被引 21

用混合方法压缩视觉语言模型的视觉令牌,提升效率与性能。

Hybrid Token Compression for Vision-Language Models

  • 分离语义与外观,连续路径保细节,离散路径用4个锚点提语义。
  • 仅用1个输出令牌,在7个基准上达87.2%性能保留,优于现有方法。
  • 适合追求高效推理的视觉语言模型部署者,尤其关注压缩与精度平衡。

视觉语言模型(VLMs)依赖数百个视觉令牌,导致计算和内存开销高。现有压缩方法存在权衡:连续压缩会削弱高层语义,离散量化可能丢失细粒度外观细节。本文提出HTC-VLM,一种通过两条互补路径解耦语义与外观的混合视觉令牌压缩框架。连续路径保留ViT块特征,离散路径使用MGVQ量化生成4个语义锚点。两条路径融合为580个令牌的混合序列,再通过解耦注意力掩码和<voco>瓶颈压缩为单个令牌。在相同单令牌输出预算下,HTC-VLM在七个基准(GQA、VQAv2、MMBench、MME、POPE、SEED-Bench和ScienceQA-Image)上平均性能保留率达87.2%,显著优于领先连续基线(81.0%)。注意力分析显示压缩令牌优先关注离散锚点,验证其作为语义引导的作用。进一步研究了令牌预算扩展、跨架构泛化、推理效率、码本与掩码变化鲁棒性及混合瓶颈的信息论特性。结果表明,结合连续外观特征与离散语义锚点可实现高效极端视觉令牌压缩。

原文摘要 · Abstract (English)

Vision-language models (VLMs) rely on hundreds of visual tokens, leading to high computational and memory costs. Existing compression methods face a trade-off: continuous compression can weaken high-level semantics, while discrete quantization may lose fine-grained appearance details. We introduce HTC-VLM, a hybrid visual token compression framework that disentangles semantics and appearance through two complementary pathways. A continuous pathway preserves detailed ViT patch features, while a discrete pathway provides semantic anchors using MGVQ quantization represented by four tokens. The two pathways are fused into a 580-token hybrid sequence and compressed into a single token using a disentanglement attention mask and a <voco> bottleneck. Under the same one-token output budget, HTC-VLM achieves 87.2% average performance retention across seven benchmarks (GQA, VQAv2, MMBench, MME, POPE, SEED-Bench, and ScienceQA-Image), outperforming the leading continuous baseline at 81.0%. Attention analysis shows that the compressed token prioritizes discrete anchors, supporting their role as semantic guidance. We further study token-budget scaling, cross-architecture generalization, inference efficiency, robustness to codebook and masking variations, and the information-theoretic properties of the hybrid bottleneck. These results show that combining continuous appearance features with discrete semantic anchors enables effective extreme visual token compression for efficient VLMs.

视觉语言模型令牌压缩混合机制高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。