让机器人听懂人话并记住经验,用低延迟实现跨机协作导航。
A Semantic Autonomy Framework for VLM-Integrated Indoor Mobile Robots: Hybrid Deterministic Reasoning and Cross-Robot Adaptive Memory
- 混合推理架构:简单指令无需调用大模型,0.1毫秒内完成
- 跨机器人记忆共享,延迟降低10.3万倍,准确率达100%
- 仅需树莓派5即可运行,零训练数据,适合真实场景部署
自主室内移动机器人虽能基于ROS 2 Navigation 2可靠导航至具体坐标,却难以理解表达意图的自然语言指令。视觉语言模型(VLM)可提供语义推理能力,但其在消费级硬件上单次决策延迟达2-9秒,且存在会话间遗忘问题,限制实际应用。本文提出语义自主栈(Semantic Autonomy Stack),一个六层参考框架,并在物理机器人上验证了包含混合确定性-VLM推理与跨机器人自适应记忆的完整实例。七步参数化解析器可在不调用语言模型、摄像头或GPU的情况下,处理88%的指令,耗时低于0.1毫秒;仅对真正模糊的指令才启用VLM推理。五类语义记忆框架结合显式作用域分类(全局环境知识、每位操作员偏好、每台机器人能力),支持跨会话学习与跨机器人知识迁移:通过一次机器人的VLM交互所学偏好,经编译摘要后传递至另一机器人,实现103,000倍的延迟降低。在两台自研差速驱动机器人上,针对82个场景级决策、三轮会话的实验表明,语义传递准确率100%(33/33,95%置信区间[0.894, 1.000]),语义解析准确率100%,且支持多机器人并发运行——所有实验均在无独立GPU的树莓派5平台上完成,无需任何训练数据。
原文摘要 · Abstract (English)
Autonomous indoor mobile robots can navigate reliably to metric coordinates using established frameworks such as ROS 2 Navigation 2, yet they lack the ability to interpret natural language instructions that express intent rather than positions. Vision-Language Models offer the semantic reasoning required to bridge this gap, but their inference latency (2-9 seconds per decision on consumer hardware) and session-by-session amnesia limit practical deployment. This paper presents the Semantic Autonomy Stack, a six-layer reference framework for semantically autonomous indoor navigation, and validates a complete instance featuring hybrid deterministic-VLM reasoning and cross-robot adaptive memory on physical robots with off-the-shelf edge hardware. A seven-step parametric resolver handles 88% of instructions in under 0.1 milliseconds without invoking a language model, camera, or GPU; only genuinely ambiguous instructions escalate to VLM reasoning. A five-category semantic memory framework with explicit scope taxonomy (global environment knowledge, per-operator preferences, per-robot capabilities) enables cross-session learning and cross-robot knowledge transfer: preferences learned through VLM interactions on one robot are promoted to deterministic resolution and transferred to a second robot via a shared compiled digest, achieving a measured latency reduction of 103,000-fold. Experimental validation on two custom-built differential-drive robots across 82 scenario-level decisions and three sessions demonstrates 100% semantic transfer accuracy (33/33, 95% CI [0.894, 1.000]), 100% semantic resolution accuracy, and concurrent multi-robot operation feasibility - all on Raspberry Pi 5 platforms with no onboard GPU, requiring zero training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。