用自然语言指挥机器人团队,让它们实时协作完成复杂任务。
Prompting Robot Teams with Natural Language
- 将语言模型的推理能力转为小型RNN,实现离线部署
- 通过图神经网络解析RNN状态,支持分布式实时控制
- 在仿真与真实场景中验证了高效、鲁棒的团队协作能力
本文提出一种框架,通过自然语言指令指挥多机器人团队完成高层次任务。目标是利用语言模型的推理能力理解并分解多机器人协作与决策任务,但在部署时无法调用语言模型的情况下。由于难以从团队指令中精确定义单个机器人的行为,并使其持续适应其他机器人的动作,因此需要一个具备任务逻辑与语义表达能力,同时支持去中心化、实时运行的框架。我们通过识别任务可表示为确定性有限自动机,并利用循环神经网络(RNN)编码大量自动机,将语言模型获得的任务逻辑与子任务序列分解结果提炼为RNN模型,使其内部状态对齐任务语义。这使得模型体积极小,可嵌入机载系统。为实现去中心化执行,我们训练一个基于RNN隐状态和语言嵌入的图神经网络控制策略。在模拟和真实世界多机器人任务中进行评估,任务要求团队具备顺序性和协作性行为,结果表明该方法具备可扩展性、鲁棒性和实时性能。详情见:sites.google.com/view/prompting-teams。
原文摘要 · Abstract (English)
This paper presents a framework to prompt multi-robot teams with high-level tasks using natural language expressions. Our objective is to use the reasoning capabilities of language models in understanding and decomposing multi-robot collaboration and decision-making tasks, but in settings where such models cannot be called at deployment time. However, it is hard to specify the behavior of an individual robot from a team instruction, and have it continuously adapt to actions from other robots. This necessitates a framework with the representational capacity required by the logic and semantics of a task, and yet supports decentralized, real-time operation. We solve this dilemma by recognizing that a task can be represented as a deterministic finite automaton, and that recurrent neural networks (RNNs) can encode numerous automata. This allows us to distill the logic and sequential decompositions of sub-tasks obtained from a language model into an RNN, and align its internal states with the semantics of a given task. This leads to a tiny model that encapsulates the reasoning of the language model and can be implemented onboard. To interpret the internal state of the RNN for a decentralized execution, we train a graph neural network control policy conditioned on the hidden states of the RNN and the language embeddings. We present evaluations on simulated and real-world multi-robot tasks that require sequential and collaborative behavior by the team, demonstrating scalable, robust, real-time performance -- sites.google.com/view/prompting-teams.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。