通过关注未来层语义注意力，实现视觉-语言-动作模型高效推理
SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation
- 基于未来层注意力一致性预测关键视觉标记
- 实测加速1.89倍，成功率下降不足1.5%
- 适合需要实时推理的机器人控制场景
实时推理视觉-语言-动作（VLA）模型对机器人控制至关重要。尽管视觉标记剪枝在加速推理方面展现出潜力，但现有方法多依赖浅层特征做剪枝决策，可能误删深层所需信息。为此，我们提出SAFE-Pruner，一个即插即用的剪枝框架，将未来层的注意力线索引入剪枝判断。我们发现，VLA模型在不同控制时间步间倾向于将注意力集中在同一语义实体上，称为语义注意力一致性。基于此，设计前瞻策略预测深层的标记重要性，防止过早移除关键信息，提升加速稳定性。进一步提出参考时间步刷新机制，在注意力转移时触发更新，提高预测精度与剪枝可靠性。在多种评估设置下，实验表明该方法可实现最高1.89倍加速，成功率下降小于1.5%，优于当前最优方法最多1.9倍。
原文摘要 · Abstract (English)
Real-time inference of vision-language-action (VLA) models is essential for robotic control. While visual token pruning has shown strong potential for accelerating inference, most existing methods mainly base pruning decisions on shallow-layer cues and risk discarding visual information required by deep layers. To address this issue, we propose SAFE-Pruner, a plug-and-play pruning framework that incorporates attention cues of future layers into pruning decisions. Specifically, we identify semantic attention consistency, the tendency that VLA models concentrate their attention probability mass on the same semantic entity across control timesteps. Based on this observation, we design a forward-looking strategy to forecast the token saliency in deep layers, which prevents the premature removal of critical tokens and leads to more stable acceleration. We further introduce a reference timestep refresh strategy that triggers updates upon attention shifts, thereby improving forecasting accuracy and pruning reliability. Extensive experiments across diverse evaluation settings demonstrate that our method achieves up to 1.89x speedup with a minimal degradation in success rate of less than 1.5%, while outperforming state-of-the-art methods by up to 1.9%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。