通过聚类与去噪,让视觉语言模型在极低算力下仍能抗干扰、高精度运行。
Clustering and Token Denoising for Faster and More Robust VLMs

- 用注意力加权聚类选代表性视觉标记,再一通量去噪,无需训练
- 97%压缩视觉标记至16个,极端噪声下性能仍比现有方法高20%
- 适合边缘设备部署,尤其对噪声敏感或算力受限场景
近期视觉语言模型(VLMs)通过在预训练语言模型中加入视觉标记,显著提升了多模态能力,如LLaVA表现优异。然而,处理多达576或729个视觉标记的计算负担,使边缘部署困难。尽管已有多种标记剪枝技术,但多数需重新训练;部分无训练方法可轻松适应架构变化。本文提出ClustRS,一种两阶段、无需训练的鲁棒标记剪枝算法。第一阶段为注意力加权聚类,从每个语义簇中选取代表性标记;第二阶段为残差收缩,对选中的标记进行单次去噪。这两项轻量级无训练步骤使LLaVA适用于真实数据,在多种图像噪声类型和强度下均提升鲁棒性。在ScienceQA-IMG和MM-VET基准测试中,本方法在极端噪声与标记压缩条件下(标记减少97%,仅剩16个),相比基于注意力和多样性的方法性能提升最高达20%;在LLaVA-OneVision上,于轻微噪声下以不到三分之一的标记数量达到基线性能。研究证明,该方法为仅基于得分或多样性的剪枝规则提供了简单而强大的替代方案,推动了高效且抗噪声的VLM部署。
原文摘要 · Abstract (English)
Recent Visual-Language Models (VLMs) have enhanced the capabilities of pre-trained LLMs by adding vision tokens alongside text, with approaches like LLaVA showing impressive results. However, the computational burden of processing up to 576 or 729 visual tokens makes edge deployment challenging. While various token pruning techniques require retraining, some are training-free and thus can easily adapt to architecture changes. We introduce ClustRS, a two-part, training-free algorithm for robust token pruning. Its first component is an attention-weighted, clustering algorithm that selects representative tokens from each semantic cluster. The second component, Residual Shrinkage, is a one-pass denoising step on the selected tokens. These training-free lightweight steps make LLaVA ready for real-world data, improving robustness to a wide range of image-noise types and intensities. Experimental results on the ScienceQA-IMG and MM-VET benchmarks show our method outperforms attention- and diversity-based methods by up to 20\% under extreme noise and token conditions (reducing tokens by 97\%, down to 16 tokens) on LLaVA 1.5 7b and achieves exceptional results on LLaVA-OneVision, where we match baseline performance with fewer than one-third of their tokens under mild noise conditions. Our study demonstrates a simple yet powerful alternative to both score-only and diversity-only pruning rules, paving the way for compute-efficient and noise-resilient VLM deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。