轻量级多模态模型实现嵌入式机器人本地实时控制
LiteVLA-Edge: Quantized On-Device Multimodal Control for Embedded Robotics
- FP32微调后采用4比特量化，结合GPU加速推理
- 端到端延迟仅150.5毫秒，达6.6赫兹实时性
- 适合需要离线部署的嵌入式机器人系统
视觉-语言-动作（VLA）模型为感知、语言引导与动作生成提供了统一框架，但现有系统因计算需求高和推理延迟大，难以在嵌入式机器人环境中部署。本文提出LiteVLA-Edge，一种面向部署的VLA流水线，可在Jetson Orin类硬件上实现完全本地化推理。方法结合监督图像到动作的FP32微调、训练后4比特GGUF量化及通过llama.cpp运行时的GPU加速推理。在该部署配置下，LiteVLA-Edge实现平均端到端延迟150.5毫秒（约6.6赫兹），且完全离线运行于集成ROS 2的感知-推理-执行管道中。本工作未引入新策略目标，而是提供了一条实用的系统路径，使紧凑的多模态控制模型可本地运行于嵌入式设备，同时保持感知、推理与执行模块间的解耦接口。结果验证了反应式语言控制的时间可行性，并为未来机器人任务级本地化VLA评估提供了可复现基准。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models provide a unified framework for perception, language conditioning, and action generation, but many existing systems remain difficult to deploy in embedded robotic settings because of their computational requirements and inference latency. In this paper, we present LiteVLA-Edge, a deployment-oriented VLA pipeline for fully on-device inference on Jetson Orin-class hardware. Our approach combines supervised image-to-action fine-tuning in FP32 with post-training 4-bit GGUF quantization and GPU-accelerated inference through the \texttt{llama.cpp} runtime. Under our deployment configuration, LiteVLA-Edge achieves a mean end-to-end latency of 150.5\,ms (approximately 6.6\,Hz) while operating entirely offline within a ROS~2-integrated perception--reasoning--action pipeline. Rather than introducing a new policy objective, our contribution is a practical systems path for executing compact multimodal control models locally on embedded hardware while preserving modular interfaces between perception, reasoning, and actuation. These results establish timing feasibility for reactive language-conditioned control and provide a reproducible baseline for future task-level evaluation of on-device VLAs in robotics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。