动态剪枝提升多视角视觉语言模型推理速度,兼顾精度与任务相关性。
MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Driving

- 分两阶段动态分配剪枝预算,依据视图信息多样性与指令引导选择关键令牌
- 在DriveLM上实现87.3%计算量减少、4.97倍预填充加速,精度保留98.5%
- 适用于自动驾驶中对实时性与语义理解要求高的多视角视觉语言任务
视觉语言模型(VLMs)虽提升了自动驾驶中的泛化与可解释性,但因长视觉令牌序列导致效率低下,尤其在标准多视角设置下更为显著。现有剪枝方法采用固定剪枝率和静态重要性度量,忽略视图间动态重要性差异及推理过程中信息重要性的演变。我们的分析表明,多视角VLM在深层隐含任务相关视图先验,并表现出动态信息需求。受此启发,我们提出MVPruner,一种两阶段自适应令牌剪枝方法,使剪枝行为匹配模型的动态信息需求。第一阶段根据各视图的信息多样性分配剪枝预算,并保留跨阶段贡献一致的令牌,确保语义表征能力;第二阶段基于指令文本引导预算分配与令牌选择,保证任务对齐。在四个基准测试上的实验结果表明,该方法性能优越。例如,搭载MVPruner的DriveMM在DriveLM基准上实现87.3%的FLOPs降低、预填充阶段4.97倍加速,同时保持98.5%的准确率。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) improve generalization and interpretability in autonomous driving but suffer from efficiency issues due to long visual token sequences, particularly in standard multi-view settings. Existing token pruning methods employ fixed pruning rate allocation and static importance metrics, ignoring dynamic inter-view importance differences and the evolving information importance during inference. Our analysis reveals that multi-view VLMs inherently encode task-related view priors in deeper layers and exhibit dynamic information requirements. Motivated by these findings, we propose MVPruner, a two-stage adaptive token pruning method that aligns pruning behavior with the model's dynamic information requirements. The first stage allocates pruning budgets based on the information diversity of each view, and retains tokens with consistent contribution across stages, ensuring semantic representational capacity. The second stage allocates budgets and selects tokens guided by instruction text to guarantee task alignment. Experimental results on four benchmarks demonstrate the superior performance of our method. For example, DriveMM equipped with MVPruner achieves 87.3% reduction in FLOPs, 4.97* speedup in prefilling phase while retaining 98.5% accuracy on DriveLM benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。