通过一致性保持剪枝与延迟修复,大幅减少视觉语言模型推理开销。
RCP: Representation Consistency Pruner for Mitigating Distribution Shift in Large Vision-Language Models
- 基于注意力机制预测累积裁剪掩码,实现层间一致的视觉标记逐步移除
- 可删减88.9%视觉标记,降低85.7%计算量,准确率损失极小
- 无需微调主模型,适合部署在资源受限的视觉语言系统中
大型视觉语言模型(LVLM)因语言解码器处理海量视觉标记而产生高昂推理成本。现有剪枝方法常导致性能显著下降,因其不可逆地移除视觉标记会引发隐藏状态分布偏移,偏离预训练时的全标记状态。为此,我们提出表示一致性剪枝器(RCP),一种融合累积视觉标记剪枝与延迟修复机制的新框架。具体而言,我们设计了一种跨注意力剪枝器,利用大语言模型的内在注意力作为基线,预测累积掩码,确保各层间令牌减少的一致性与单调性。为补偿信息损失,我们引入延迟修复适配器(DRA),缓存被剪枝标记的核心特征,并对答案生成标记进行基于FiLM的调制。采用修复损失,使剪枝后表示的一阶与二阶统计特性与全标记教师模型匹配。RCP高效且轻量,仅需训练插件模块,推理时可实际丢弃标记。大量实验表明,RCP最多可移除88.9%的视觉标记,降低85.7%的浮点运算量,平均准确率下降极小,优于无需微调原模型的现有方法,在多个主流基准上表现更优。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) suffer from prohibitive inference costs due to the massive number of visual tokens processed by the language decoder. Existing pruning methods often lead to significant performance degradation because the irreversible removal of visual tokens causes a distribution shift in the hidden states that deviates from the pre-trained full-token regime. To address this, we propose Representation Consistency Pruner, which we refer to as RCP, as a novel framework that integrates cumulative visual token pruning with a delayed repair mechanism. Specifically, we introduce a cross-attention pruner that leverages the intrinsic attention of the LLM as a baseline to predict cumulative masks, ensuring consistent and monotonic token reduction across layers. To compensate for the resulting information loss, we design a delayed repair adapter denoted as DRA, which caches the essence of pruned tokens and applies FiLM-based modulation specifically to the answer generation tokens. We employ a repair loss to match the first and second-order statistics of the pruned representations with a full-token teacher. RCP is highly efficient because it trains only lightweight plug-in modules while allowing for physical token discarding at inference. Extensive experiments on LVLM benchmarks demonstrate that RCP removes up to 88.9\% of visual tokens and reduces FLOPs by up to 85.7\% with only a marginal average accuracy drop, and outperforms prior methods that avoid fine-tuning the original model on several widely used benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。