让视觉语言动作模型学会动态缓存视觉令牌,提速同时提升成功率。
Learning to Accelerate Vision-Language-Action Models through Adaptive Visual Token Caching
- 设计可学习的缓存决策机制,根据任务动态选择复用哪些视觉令牌。
- 在LIBERO上实现1.76倍推理加速,成功率提升1.9个百分点至76.9%。
- 适合追求高效高精度机器人控制的开发者与研究者参考。
视觉-语言-动作(VLA)模型在机器人操作任务中展现出强大的泛化能力,但其巨大的计算开销仍是实际部署的关键障碍。提升推理效率对实际应用至关重要。现有加速方法多依赖启发式或静态策略(如基于规则的令牌缓存或剪枝),与任务目标解耦,无法适应动态场景变化。本文将推理加速重构为可学习的策略优化问题,提出一种新型框架,将动态、任务感知的决策过程直接嵌入VLA模型。核心包含两个轻量级协同模块:缓存令牌选择器,决定哪些令牌应被复用;缓存比例预测器,控制复用令牌的数量。由于决策具有离散性,训练困难,我们采用可微松弛方法,实现端到端梯度优化。在LIBERO和SIMPLER基准上的大量实验及真实机器人评估表明,该方法在保持性能的同时实现1.76倍的时钟推理加速,并在LIBERO上平均成功率从75.0%提升至76.9%(+1.9个百分点),在真实任务中提升5.0个百分点,显著优于现有基线。本工作展示了学习任务感知计算分配策略的潜力,为兼具强大能力与高效率的VLA模型铺平道路。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models have demonstrated remarkable generalization capabilities in robotic manipulation tasks, yet their substantial computational overhead remains a critical obstacle to real-world deployment. Improving inference efficiency is therefore essential for practical robotic applications. Existing acceleration methods often rely on heuristic or static strategies--such as rule-based token caching or pruning--that are decoupled from task objectives and fail to adapt to dynamic scene changes. In this work, we reformulate inference acceleration as a learnable policy optimization problem and propose a novel framework that integrates a dynamic, task-aware decision-making process directly into the VLA model. At its core are two lightweight, cooperative modules: a Cached Token Selector, which determines which tokens should be reused, and a Cache Ratio Predictor, which controls how many tokens to reuse. Training these modules is non-trivial due to their discrete decisions. We address this by adopting a differentiable relaxation that allows gradient-based end-to-end optimization. Extensive experiments on the LIBERO and SIMPLER benchmarks, as well as real-robot evaluations, show that our method achieves a 1.76x wall-clock inference speedup while simultaneously improving the average success rate by 1.9 percentage points (from 75.0% to 76.9%) on LIBERO and by 5.0 percentage points on real-world tasks, significantly outperforming existing baselines. This work highlights the potential of learning task-aware computational allocation policies, paving the way for VLA models that are both powerful and efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。