用可解释的概念重叠来智能删减视觉令牌,提升视觉语言模型效率。
TORINO: Token Reduction via Interpretable Concept Overlap in Vision-Language Models

- 通过稀疏自编码器将视觉令牌映射到可解释的语义空间,按共享概念分组。
- 动态调整删减比例,复杂图像保留更多令牌,性能损失小于1%。
- 无需微调模型,适配多场景,适合部署在资源受限设备上。
视觉语言模型(VLMs)在多项任务中表现优异,但其计算开销主要来自输入语言模型的大量视觉令牌。现有令牌缩减方法依赖注意力得分或成对相似性,缺乏每个令牌的显式语义表示。本文提出TORINO(TOken Reduction via Interpretable coNcept Overlap),一种无需微调底层模型的即插即用框架,实现VLM中自适应视觉令牌缩减。TORINO利用稀疏自编码器(SAEs)将视觉令牌投影至可解释的潜在空间,通过共享概念激活分析令牌间关系。我们定义概念重叠为活跃SAE潜变量间的共识程度,并据此分组共享语义内容的令牌。组内采用剪枝或合并进行缩减,统一保留关键语义信息并去除冗余。不同于固定预算方法,TORINO根据输入复杂度动态调节缩减率,使不同图像保留不同数量的令牌。在多个视觉语言基准上的实验表明,TORINO在保持性能的前提下显著减少视觉令牌数,实现了良好的效率-精度权衡。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have demonstrated impressive capabilities across different tasks, but their computational cost is dominated by the large number of visual tokens fed to the language model. Existing token reduction methods rely on attention-based scores or pairwise similarity, without an explicit semantic representation of each token. We introduce TORINO (TOken Reduction via Interpretable coNcept Overlap), a plug-and-play framework for adaptive visual token reduction in VLMs that requires no fine-tuning of the underlying model. TORINO leverages Sparse Autoencoders (SAEs) to project visual tokens into an interpretable latent space where token relationships can be analyzed through shared concept activations. Specifically, we define concept overlap as the degree of agreement between active SAE latents and use it to group tokens that share semantic content. Reduction within each group is then performed by either pruning or merging, providing a unified framework that preserves semantically important visual information while removing redundancy. Unlike fixed-budget approaches, TORINO dynamically adapts the reduction rate to input complexity, allowing different images to retain different numbers of tokens. Experiments across multiple vision-language benchmarks show that TORINO achieves favorable efficiency-accuracy trade-offs, reducing the number of visual tokens with minimal performance loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。