通过低秩结构识别关键视觉令牌,显著减少计算量而不损失性能。
LRCP: Low-Rank Compressibility Guided Visual Token Pruning for Efficient LVLMs

- 基于主成分分析提取视觉令牌的低秩主子空间,用投影残差评估重要性。
- 图像任务保留94.7%性能,仅需11.1%令牌;视频任务保持97.8%准确率,令牌减少87.5%。
- 无需训练、通用性强,适合高分辨率图像与长视频的高效推理场景。
大型视觉语言模型(LVLMs)虽具备强大多模态理解能力,但其推理成本随视觉令牌数量快速上升,尤其在高分辨率图像和长视频上。现有基于注意力的方法依赖注意力分数评估令牌重要性,可能引入位置偏差;基于表示的方法则依据特征关系或重构误差减少冗余,忽视视觉令牌集合的全局结构。本文从低秩可压缩性视角重新审视视觉令牌压缩,在多个模型与数据集上发现视觉令牌表征具有显著低秩特性,即使移除大量随机令牌,主导子空间仍保持稳定。受此启发,我们提出无训练压缩框架LRCP:先通过PCA估计视觉令牌的主导低秩子空间,再以每个令牌在该子空间上的投影残差作为评分,保留被低秩背景难以解释的令牌。大量实验表明,LRCP在图像任务中仅用11.1%的令牌即可保持94.7%的原始理解性能,视频任务中在87.5%的令牌缩减下维持97.8%的平均理解准确率。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) achieve strong multimodal understanding, but their inference cost grows rapidly with the number of visual tokens, especially for high-resolution images and long videos. Existing attention-based methods estimate token importance from attention scores, which may introduce positional bias, while representation-based methods reduce visual redundancy based on feature relations or reconstruction errors, overlooking the global structure of the visual token set. In this paper, we revisit visual token compression from the perspective of low-rank compressibility. Across models and datasets, we observe that visual token representations exhibit a pronounced low-rank structure, with a dominant subspace that remains stable even after a large fraction of tokens is randomly removed. Motivated by this finding, we propose LRCP, a training-free compression framework that first estimates the dominant low-rank subspace of visual tokens via PCA, and then scores each token by its projection residual onto this subspace, retaining tokens that are poorly explained by the low-rank background. Extensive experiments show that LRCP achieves superior results, preserving 94.7% of the original image-understanding performance with an 88.9% token reduction and 97.8% of the average video-understanding accuracy with an 87.5% token reduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。