单卡实现实时运行多视角视觉语言模型，30帧/秒控制机器人抓取掉落钢笔。
Running VLAs at Real-time Speed
- 通过优化推理流程消除延迟，实现低延迟高效计算。
- 在真实任务中达成100%抓取成功率，动态响应速度达480Hz。
- 适合需要实时交互的机器人控制场景，如智能抓取与导航。
本文展示如何在单张消费级显卡上以30帧/秒的帧率、最高480Hz的轨迹频率运行pi0级别多视角视觉语言模型（VLA），使此前认为大型VLA无法实现的动态实时任务成为可能。为此，我们提出一系列策略以消除模型推理中的开销。真实实验表明，采用该策略的pi0策略在抓取掉落钢笔任务中实现了100%的成功率。基于此，我们进一步提出了完整的流式推理框架，用于支持VLA在机器人实时控制中的应用。代码已开源：https://github.com/Dexmal/realtime-vla。
原文摘要 · Abstract (English)
In this paper, we show how to run pi0-level multi-view VLA at 30Hz frame rate and at most 480Hz trajectory frequency using a single consumer GPU. This enables dynamic and real-time tasks that were previously believed to be unattainable by large VLA models. To achieve it, we introduce a bag of strategies to eliminate the overheads in model inference. The real-world experiment shows that the pi0 policy with our strategy achieves a 100% success rate in grasping a falling pen task. Based on the results, we further propose a full streaming inference framework for real-time robot control of VLA. Code is available at https://github.com/Dexmal/realtime-vla.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。