将视觉语言动作模型部署到机器人端，支持多硬件平台高效运行。
vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models
- 基于llama.cpp构建C++推理引擎，原生支持流匹配与扩散型VLA推理流程。
- 在LIBERO-Object上达到99.5%成功率，仅需1.3 GiB内存即可运行BitVLA。
- 同一模型包可在消费级显卡至嵌入式设备间无修改运行，适合机器人部署场景。
视觉-语言-动作（VLA）策略通常以Python/PyTorch堆栈形式发布，依赖工作站级GPU，与实际机器人硬件不匹配。本文提出vla.cpp，一个基于llama.cpp的可移植C++推理运行时。据我们所知，这是首个原生支持流匹配与扩散型VLA推理模式的ggml类引擎，其中缓存的视觉-语言前缀由跨注意力动作专家在多个求解步骤中逐步处理。单一运行时支持七种架构，涵盖五种主干网络和四种动作头家族，通过统一请求/响应协议交互，每模型以自包含包形式封装。在LIBERO-Object数据集上，该引擎与最先进检查点相比，200次任务中仅差1次；在相同配置下，BitVLA实现100%成功率达1.3 GiB内存占用。同一模型包可在三类硬件层级（从消费级显卡到8 GB嵌入式模块）无修改运行。跨硬件吞吐分析表明，单批处理的VLA推理受计算限制，利用率是关键优化方向；基于此分析提出的IMMA阶梯型GEMM将BitVLA每步延迟降低4.5倍。进一步在ALOHA机械臂上开展机器人端压力测试，隔离出真实训练硬件下的延迟约束，验证了学习到的VLA在动态目标下必须重规划的性能边界。代码、演示视频及可复现基准框架已公开于 https://fai-modelopt-tech.github.io/vla-cpp.github.io/。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) policies are typically shipped as Python/PyTorch stacks that assume a workstation-class GPU, a mismatch for the hardware on which robots actually run. We present vla.cpp, a portable C++ inference runtime built on llama.cpp. To our knowledge, it is the first ggml-class engine to natively serve the flow-matching and diffusion VLA inference pattern, in which a cached vision-language prefix is consumed by a cross-attending action expert integrated over several solver steps. A single runtime serves seven architectures spanning five backbone and four action-head families behind one request/response protocol, with each model packaged as a self-contained bundle. On LIBERO-Object, the engine matches a state-of-the-art checkpoint to within one episode out of 200, and runs BitVLA at 100% success in 1.3 GiB of memory. The same bundle runs unchanged across three hardware tiers, from a consumer GPU down to an 8 GB embedded module. A cross-hardware roofline analysis shows that batch-1 VLA inference is compute-bound, so utilization rather than bandwidth is the deployment lever; an IMMA ladder GEMM derived from this analysis cuts BitVLA per-step latency by 4.5x. We then frame an on-robot stress test on an ALOHA arm that isolates the latency constraint under which a learned VLA must replan against a moving target on the hardware it was trained for. Code, demo videos, and the reproducible benchmark scaffold are available at https://fai-modelopt-tech.github.io/vla-cpp.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。