通过结合动作全局上下文,实现视觉-语言-动作模型的高效剪枝。
SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning
- 基于动作级静态与层级动态剪枝,融合局部与全局信息。
- 在仿真和真实任务中分别实现1.57×和1.70×加速,成功率几乎无损。
- 无需训练,适配机器人控制等实时视觉-语言任务。
剪枝是加速计算密集型模型的常用方法,通过移除不重要计算来提升效率。近期已有研究将其应用于视觉-语言-动作(VLA)模型推理加速,但现有方法仅关注当前动作步骤的局部信息,忽视全局上下文,导致某些场景下成功率下降超过20%,加速效果有限。本文指出,VLA任务中连续步骤的输入图像具有高度时空一致性,提出关键洞察:令牌选择应结合局部信息与模型全局上下文。基于此,我们提出SpecPrune-VLA,一种无需训练的两级剪枝方法,含启发式控制器。首先,动作级静态剪枝:利用全局历史与局部注意力,静态减少每动作的视觉令牌数;其次,层级动态剪枝:按层重要性自适应剪枝;第三,轻量级动作感知控制器:根据末端执行器速度将动作分类为粗粒度或细粒度,并相应调整剪枝强度。大量实验表明,SpecPrune-VLA在LIBERO仿真中实现最高1.57×加速,在真实任务中达1.70×,成功率几乎无损。
原文摘要 · Abstract (English)
Pruning is a typical acceleration technique for compute-bound models by removing computation on unimportant values. Recently, it has been applied to accelerate Vision-Language-Action (VLA) model inference. However, existing acceleration methods focus on local information from the current action step and ignore the global context, leading to >20% success rate drop and limited speedup in some scenarios. In this paper, we point out spatial-temporal consistency in VLA tasks: input images in consecutive steps exhibit high similarity, and propose the key insight that token selection should combine local information with global context of the model. Based on this, we propose SpecPrune-VLA, a training-free, two-level pruning method with heuristic control. (1) Action-level static pruning. We leverage global history and local attention to statically reduce visual tokens per action. (2) Layer-level dynamic pruning. We prune tokens adaptively per layer based on layer-wise importance. (3) Lightweight action-aware controller: We classify actions as coarse- or fine-grained by the speed of the end effector and adjust pruning aggressiveness accordingly. Extensive experiments show that SpecPrune-VLA achieves up to 1.57$\times$ speedup in LIBERO simulation and 1.70$\times$ on real-world tasks, with negligible success rate degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。