arXiv:2508.13305cs.CV2025-08中稿 · CVPR被引 16

提出轻量级视觉令牌剪枝框架,让自动驾驶多视角大模型更快更省显存。

Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving

  • 按语义与空间多样性筛选关键视觉令牌,避免信息丢失。
  • 仅保留10%令牌时,推理速度提升6.4倍,计算量降至13.4%。
  • 无需重训练或修改模型,适配现有高效注意力结构。

视觉语言模型(VLM)在自动驾驶中展现出统一感知与决策的潜力,但高分辨率多视角图像处理带来巨大计算开销,源于视觉令牌数量庞大及自注意力的二次复杂度。为此,我们提出Prune2Drive——一种即插即用的多视角VLM视觉令牌剪枝框架。该框架包含两项核心创新:(i) 基于多样性的令牌选择机制,确保跨视角语义与空间覆盖;(ii) 视角自适应剪枝控制器,自动学习各摄像头对下游任务的重要性并确定最优剪枝率。相比以往方法,Prune2Drive无需模型重训练或访问注意力图,兼容现代高效注意力实现。在DriveLM和DriveLMM-o1基准上实验证明,仅保留10%视觉令牌时,预填充阶段速度提升6.40倍,仅消耗原模型13.4%的浮点运算量,且在DriveLM上平均性能仅下降3%。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have emerged as a promising paradigm in autonomous driving (AD), providing a unified framework for perception and decision-making. However, their real-world deployment is hindered by significant computational overhead when processing high-resolution, multi-view images. This complexity stems from the massive number of visual tokens, which increases inference latency and memory consumption due to the quadratic complexity of self-attention. To address these challenges, we propose Prune2Drive, a plug-and-play visual token pruning framework for multi-view VLMs in AD. Prune2Drive introduces two core innovations: (i) a diversity-aware token selection mechanism that prioritizes semantic and spatial coverage across views, and (ii) a view-adaptive pruning controller that automatically learns optimal pruning ratios based on camera importance to downstream tasks. Unlike prior methods, Prune2Drive requires no model retraining or access to attention maps, ensuring compatibility with modern efficient attention implementations. Extensive experiments on the DriveLM and DriveLMM-o1 benchmarks demonstrate that Prune2Drive achieves significant speedups and memory savings with minimal performance impact. When retaining only 10% of visual tokens, our method achieves a 6.40x speedup in the prefilling phase and consumes only 13.4% of the original FLOPs, with a mere 3% average performance drop on the DriveLM benchmark. Code is available at: https://github.com/MinhaoXiong/Prune2Drive.git

自动驾驶视觉语言模型剪枝多视角

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。