让视觉模型自动删掉不重要的图像信息,又快又准。
The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruning
- 用可微分方法动态判断图像片段重要性,自动剪枝
- 在LIBERO任务上降低59%算力、38%延迟,成功率提升2.6%
- 无需额外参数,适合部署到机器人等资源受限设备
我们提出LightVLA,一种针对视觉-语言-动作(VLA)模型的可微分视觉标记剪枝框架。尽管VLA模型在执行真实世界机器人任务中表现出色,但其在资源受限平台上的部署常受制于大量视觉标记带来的高注意力计算开销。LightVLA通过自适应、性能驱动的视觉标记剪枝来解决此问题:生成动态查询评估视觉标记重要性,并采用Gumbel softmax实现可微分标记选择。经过微调,LightVLA学会保留对任务执行最有贡献的视觉标记,同时剪除无关标记,从而在提升性能的同时增强效率。该方法无需启发式超参数,且不引入额外可训练参数,兼容现代推理框架。实验表明,LightVLA在LIBERO基准上优于多种VLA模型及现有剪枝方法,在多个任务中实现更高成功率并显著降低计算开销:FLOPs减少59.1%,延迟降低38.2%,成功率提升2.6%。此外,我们也研究了带可学习查询的LightVLA*,同样取得良好表现。我们的工作揭示,当VLA追求最优性能时,LightVLA能自发地从性能驱动角度进行标记剪枝。据我们所知,LightVLA是首个将自适应视觉标记剪枝应用于VLA任务并兼顾效率与性能的工作,为更高效、强大且实用的实时机器人系统迈出了关键一步。
原文摘要 · Abstract (English)
We present LightVLA, a simple yet effective differentiable token pruning framework for vision-language-action (VLA) models. While VLA models have shown impressive capability in executing real-world robotic tasks, their deployment on resource-constrained platforms is often bottlenecked by the heavy attention-based computation over large sets of visual tokens. LightVLA addresses this challenge through adaptive, performance-driven pruning of visual tokens: It generates dynamic queries to evaluate visual token importance, and adopts Gumbel softmax to enable differentiable token selection. Through fine-tuning, LightVLA learns to preserve the most informative visual tokens while pruning tokens which do not contribute to task execution, thereby improving efficiency and performance simultaneously. Notably, LightVLA requires no heuristic magic numbers and introduces no additional trainable parameters, making it compatible with modern inference frameworks. Experimental results demonstrate that LightVLA outperforms different VLA models and existing token pruning methods across diverse tasks on the LIBERO benchmark, achieving higher success rates with substantially reduced computational overhead. Specifically, LightVLA reduces FLOPs and latency by 59.1% and 38.2% respectively, with a 2.6% improvement in task success rate. Meanwhile, we also investigate the learnable query-based token pruning method LightVLA* with additional trainable parameters, which also achieves satisfactory performance. Our work reveals that as VLA pursues optimal performance, LightVLA spontaneously learns to prune tokens from a performance-driven perspective. To the best of our knowledge, LightVLA is the first work to apply adaptive visual token pruning to VLA tasks with the collateral goals of efficiency and performance, marking a significant step toward more efficient, powerful and practical real-time robotic systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。