通过时空合并减少视觉令牌,让机器人模型更快响应。
Fast Enough to Act: Spatio-Temporal Visual Token Merging for Low-Latency Robotic VLMs and VLAs

- 在视觉编码阶段直接合并冗余视觉令牌,无需训练。
- 在1024×1024分辨率下实现8.3倍推理加速,成功率不变。
- 适合对实时性要求高的机器人视觉语言模型应用。
视觉语言模型和视觉语言动作模型赋予机器人前所未有的能力。然而,视频和高分辨率图像输入会产生大量视觉令牌,导致极高的推理延迟,严重阻碍机器人的实时控制。为突破这一计算瓶颈,我们提出ST-Merge,一种即插即用、无需训练的框架,在视觉编码阶段直接融合冗余令牌。通过显式构建三维时空坐标,采用多队列并行匹配与加权聚合机制,实现跨帧高效且几何一致的令牌融合。此外,引入后融合位置修正机制,动态重评融合中心的旋转位置编码,有效消除合并带来的空间偏差,保障精细操作所需的空间精度。在主流VLM Qwen2.5-VL的视频问答任务中,ST-Merge实现2倍推理加速,仅损失1%精度。部署于$π_{0.5}$ VLA策略时,在1024×1024分辨率下实现8.3倍加速,成功率与基线相当;低分辨率下精度略有下降。
原文摘要 · Abstract (English)
Vision-language models and vision-language action models endow the robot with unprecedented capabilities. However, the input of video and high-resolution images yields a massive number of visual tokens, leading to extremely high inference latency and severely hindering the robot's real-time control. To break through this computational bottleneck, we propose ST-Merge, a plug-and-play, training-free framework that efficiently fuses redundant tokens directly during the visual encoding phase. By explicitly constructing 3D spatiotemporal coordinates, it employs a multi-queue parallel matching and weighted aggregation mechanism to achieve efficient and geometrically consistent fusion of redundant tokens across frames. In addition, we introduce a post-merge positional correction mechanism that effectively eliminates spatial deviation caused by merging by dynamically re-evaluating the rotational position code of the weighted centroid of the vision token, thereby ensuring the high-precision spatial awareness required for dexterous operation. In the Video Question Answering task on the mainstream VLM, Qwen2.5-VL, ST-Merge achieves a 2$\times$ inference speedup with only a tiny 1\% loss in precision. When deployed on the $π_{0.5}$ VLA policy, ST-Merge achieves an 8.3$\times$ speedup at 1024 $\times$ 1024 resolution and matches the baseline success rate at this high-resolution setting. At lower resolutions, it introduces a small drop in accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。