解决视觉-语言-动作模型推理中语义与动作信息不匹配的剪枝难题
Bridging the Semantic-Action Gap in Visual Token Pruning for Efficient VLA Inference
- 融合语义重要性与动作相关性的双路径剪枝机制
- 在保持操作性能前提下实现最高1.99倍加速
- 适合需要实时响应的机器人视觉推理场景
视觉-语言-动作(VLA)模型在具身智能中展现出巨大潜力,但实时部署需处理连续视觉流,计算开销大。视觉标记剪枝是主流加速方法,但直接用于VLA推理会导致操作性能严重下降。分析发现,这是由于预填充阶段与动作解码阶段的注意力模式差异:仅基于语义显著性剪枝会误删关键动作信息。为此,我们提出VLA-Pruner,一种基于VLA推理视觉需求的即插即用剪枝方法,利用机器人操作的时间连续性,从语义预填充和时序平滑的动作相关性两方面评估标记重要性,并采用先合并后筛选策略,在计算预算内保留紧凑非冗余标记。实验表明,VLA-Pruner在多个VLA架构上优于现有方法,实现最高1.99倍加速,同时保持相当的操作质量。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models have shown great potential for embodied AI by integrating visual perception, language understanding, and action execution. In real-time deployment, these models must process continuous visual streams, incurring substantial computational overhead. Visual token pruning -- a mainstream technique for accelerating Vision-Language Models (VLMs) by retaining salient tokens while discarding redundant ones -- offers a natural candidate solution to this challenge. However, directly applying VLM-oriented pruning methods to VLA inference can cause severe degradation in manipulation performance. Our analysis attributes this degradation to a key mismatch: VLA inference exhibits distinct attention patterns between the vision-language prefill stage and the action-decode stage, so pruning based only on context-prefill semantic salience is biased toward semantic cues and may remove action-critical visual tokens. Motivated by this observation, we propose VLA-Pruner, an effective plug-and-play token pruning method grounded in the visual requirements of VLA inference, further exploiting the temporal continuity of robot manipulation. Specifically, VLA-Pruner estimates visual-token importance from both semantic prefilling and temporally smoothed action relevance, and then applies a Combine-then-Filter strategy to retain compact, non-redundant tokens under the compute budget. Experiments show that VLA-Pruner outperforms state-of-the-art approaches across multiple VLA architectures, achieving up to 1.99x speedup with comparable manipulation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。