自动寻找视觉语言模型剪枝最优配置，兼顾效率与精度。
VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models
- 将剪枝配置优化为帕累托前沿问题，用梯度搜索找最优解。
- 在8个基准上逼近网格搜索的最优权衡，通用性强。
- 发现分层渐进剪枝更符合模型压缩结构，效果更好。
视觉令牌剪枝方法能有效缓解视觉语言模型（VLMs）处理高分辨率图像和视频帧时带来的二次计算增长问题。然而，现有方法依赖预设剪枝配置，无法判断是否达到计算-性能最优。本文提出VisPCO，将视觉令牌剪枝建模为帕累托配置优化问题，通过连续松弛与直通估计实现基于梯度的搜索，采用增广拉格朗日法求解。在8个视觉基准上的大量实验表明，VisPCO能有效逼近通过网格搜索获得的实证帕累托前沿，并在多种剪枝方法与VLM架构间具有良好泛化性。此外，通过可学习核函数，我们研究了分层剪枝模式，发现多步渐进剪枝能捕捉VLM的层级压缩结构，相比单层方法取得更优的精度-效率权衡。
原文摘要 · Abstract (English)
Visual token pruning methods effectively mitigate the quadratic computational growth caused by processing high-resolution images and video frames in vision-language models (VLMs). However, existing approaches rely on predefined pruning configurations without determining whether they achieve computation-performance optimality. In this work, we introduce , a novel framework that formulates visual token pruning as a Pareto configuration optimization problem to automatically identify optimal configurations. Our approach employs continuous relaxation and straight-through estimators to enable gradient-based search, solved via the Augmented Lagrangian method. Extensive experiments across 8 visual benchmarks demonstrate that effectively approximates the empirical Pareto frontier obtained through grid search and generalizes well across various pruning methods and VLM architectures. Furthermore, through learnable kernel functions, we investigate layer-wise pruning patterns and reveal that multi-step progressive pruning captures VLMs' hierarchical compression structure, achieving superior accuracy-efficiency trade-offs compared to single-layer approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。