arXiv:2607.13500cs.CV2026-07

不依赖注意力机制,轻量级压缩视觉令牌,提升多模态模型部署效率。

Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models

论文配图:Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models
图 1 · 摘自论文原文
  • 用信息熵衡量令牌重要性,无需注意力计算
  • 通过变换一致性信号实现轻量去冗余,保持视觉多样性
  • 可直接插入主流多模态模型,适合边缘设备部署

视觉语言模型(VLMs)在多模态理解上表现强劲,但因处理大量视觉令牌而计算开销大,难以在资源受限的边缘设备上部署。令牌压缩是加速推理的可行方向,但现有方法或依赖与现代加速框架不兼容的注意力图,或依赖计算量大的成对相似性比较,削弱了可扩展性与实际部署价值。本文提出一种无注意力、轻量化的令牌压缩框架,作为即插即用模块集成于VLMs,同时保留重要与多样化的令牌以生成紧凑视觉表征。首先,为实现无注意力的重要性评估,采用信息论视角,基于新型熵准则量化令牌信息,保留特征表达力强且不退化的令牌;其次,为轻量保证视觉覆盖多样性,引入变换诱导的一致性信号,使相似令牌产生相似信号,通过排序将相似令牌聚集,结合步长采样实现多样令牌选取。在多个VLM基准测试中,该框架在极端压缩下仍保持竞争力,展现出优异的精度-效率权衡。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have achieved strong performance in multimodal understanding, yet remain challenging to deploy on resource-constrained edge devices due to the substantial computational overhead of processing numerous visual tokens. Token reduction is a promising direction for accelerating VLMs inference, but existing approaches either rely on attention maps that are incompatible with modern acceleration frameworks or depend on computationally intensive pairwise similarity comparisons, which undermine scalability and negate their practical benefits in deployment. In this paper, we propose an attention-free and lightweight token reduction framework as a plug-and-play module for VLMs, which preserves both important and diverse tokens to produce a compact visual representation. First, to enable attention-free importance estimation, we adopt an information-theoretic perspective and quantify token information using a novel entropy-based criterion, retaining those with more expressive and less degenerate feature representations. Second, to ensure diverse visual coverage in a lightweight manner, we introduce a transformation-induced consistency signal where similar tokens yield similar signals, such that sorting by this signal places similar tokens close to each other and enables stride-based selection to produce a diverse token set. Extensive experiments across multiple VLMs benchmarks demonstrate that our framework achieves a favorable accuracy-efficiency trade-off, maintaining competitive performance under aggressive compression.

视觉语言模型令牌压缩轻量化边缘部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。