将视觉语言模型部署在机器人本地，实现低成本自主控制。
Multi-Agent Robotic Control with Onboard Vision-Language Models
- 用小型化视觉语言模型构建多智能体系统，全程本地运行。
- 在仿真仓库中完成五类任务，包检准确率经微调提升显著。
- 适合需要低延迟、离线运行的工业场景，开源可复现。
视觉语言模型（VLM）和视觉语言动作模型（VLA）在机器人控制中展现出潜力，但面临可解释性差、泛化能力弱及计算资源需求高的挑战。本文提出一种多智能体系统（MAS）架构，将专用智能体部署于车载硬件，摆脱对外部计算的依赖。系统在模拟工业仓库中控制一个多用途自主移动操作机器人，完成五类任务：安全检查、仓库维护、仓库搜寻、包裹质量验证及响应人类请求。全系统采用紧凑型VLM（3-20B参数），并通过微调提升包裹检测准确率。创新设计的“Megamind”协调智能体有效缓解小模型长周期规划中的上下文记忆问题。通过搭载AMD Ryzen™ AI迷你电脑的软硬件协同仿真环境进行验证，结果表明，完全基于本地的MAS架构是云依赖方案的可行、低成本替代方案，具备良好的现实迁移潜力。仿真环境已开源，许可协议为Apache 2.0。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) and Vision Language Action (VLA) models have shown promise in robotic control. Yet, they face significant challenges regarding explainability, generalization, and compute requirements. This paper presents a Multi-Agent System (MAS) architecture that addresses these limitations by deploying specialized agents on onboard hardware - eliminating dependence on external compute. The system controls a multi-purpose autonomous mobile manipulator in a simulated industrial warehouse, fulfilling five task categories: safety inspection, warehouse maintenance, warehouse search, package quality verification, and responding to human requests. Compact VLMs (3-20B parameters) are used throughout, with fine-tuning applied to improve package inspection accuracy. A novel "Megamind" orchestration agent mitigates context retention issues inherent to long-horizon planning with smaller models. The system was validated in a hardware-in-the-loop simulation using an AMD Ryzen(TM) AI mini PC. Results demonstrate that a fully onboard MAS architecture is a viable, cost-efficient alternative to cloud-dependent deployments, with strong potential for real-world transfer. The simulation environment has been released as open source under the Apache 2.0 licence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。