通过文本引导的子空间重构，高效剪裁视觉令牌，提升大模型推理速度。
ResPrune: Text-Conditioned Subspace Reconstruction for Visual Token Pruning in Large Vision-Language Models
- 将视觉令牌剪裁转化为子空间重建问题，基于残差能量贪心选择
- 在多个基准上实现计算量与延迟显著降低，性能优于现有方法
- 无需训练、通用性强，适合部署在各类视觉语言模型中
大型视觉语言模型（LVLM）依赖密集的视觉令牌以捕捉细粒度视觉信息，但全量处理带来巨大的计算与内存开销。为此，我们提出ResPrune，一种无需训练的视觉令牌剪裁框架，通过选择紧凑且信息丰富的视觉令牌子集，实现高效的LVLM推理。ResPrune将视觉令牌剪裁建模为子空间重建问题，采用由残差能量引导的贪心子空间扩展策略，有效保留原始视觉令牌空间的几何结构。为进一步融合跨模态对齐，选择过程受文本相关性条件约束，鼓励保留既具信息量又与指令相关的令牌。该方法轻量且模型无关，可无缝集成至现有LVLM流程中，无需重训练或架构修改。在包括LLaVA-1.5、LLaVA-NeXT和Qwen2.5-VL在内的多个LVLM主干网络上进行的大量实验表明，ResPrune在多种基准上持续优于现有剪裁方法，同时实现显著的计算、内存消耗和推理延迟降低。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) rely on dense visual tokens to capture fine-grained visual information, but processing all these tokens incurs substantial computational and memory overhead during inference. To address this issue, we propose ResPrune, a training-free visual token pruning framework that enables efficient LVLM inference by selecting a compact yet informative subset of visual tokens. ResPrune formulates visual token pruning as a subspace reconstruction problem and employs a greedy subspace expansion strategy guided by residual energy, allowing it to preserve the geometric structure of the original visual token space. To further incorporate cross modal alignment, the selection process is conditioned on textual relevance, encouraging the retention of tokens that are both informative and instruction-relevant. The proposed method is lightweight and model-agnostic, and can be seamlessly integrated into existing LVLM pipelines without retraining or architectural modifications. Extensive experiments on multiple LVLM backbones, including LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL, demonstrate that ResPrune consistently outperforms existing pruning approaches across a wide range of benchmarks, while achieving effective reductions in computation, memory consumption, and inference latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。